{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"08_PT_Embeddings","provenance":[{"file_id":"https://github.com/madewithml/lessons/blob/master/notebooks/02_Basics/08_Embeddings/08_PT_Embeddings.ipynb","timestamp":1584545838024},{"file_id":"https://github.com/madewithml/lessons/blob/master/notebooks/02_Basics/11_Embeddings.ipynb","timestamp":1583247939463}],"collapsed_sections":[],"toc_visible":true},"kernelspec":{"name":"python3","display_name":"Python 3"},"accelerator":"GPU"},"cells":[{"cell_type":"markdown","metadata":{"id":"eTdCMVl9YAXw","colab_type":"text"},"source":["# Embeddings\n","\n","In this lesson we will learn how to map tokens to vectors (embeddings) that capture the contextual, semantic and syntactic value of a token in text."]},{"cell_type":"markdown","metadata":{"id":"xuabAj4PYj57","colab_type":"text"},"source":["<div align=\"left\">\n","<a href=\"https://github.com/madewithml/lessons/blob/master/notebooks/02_Basics/08_Embeddings/08_PT_Embeddings.ipynb\" role=\"button\"><img class=\"notebook-badge-image\" src=\"https://img.shields.io/static/v1?label=&amp;message=View%20On%20GitHub&amp;color=586069&amp;logo=github&amp;labelColor=2f363d\"></a>&nbsp;\n","<a href=\"https://colab.research.google.com/github/madewithml/lessons/blob/master/notebooks/02_Basics/08_Embeddings/08_PT_Embeddings.ipynb\"><img class=\"notebook-badge-image\" src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"></a>\n","</div>"]},{"cell_type":"markdown","metadata":{"id":"paO18xGs7kGX","colab_type":"text"},"source":["# Overview"]},{"cell_type":"markdown","metadata":{"id":"JqxyljU18hvt","colab_type":"text"},"source":["So far, we've also represented our text data in a one-hot encoded form where each token is represented by an n-dimensional array.\n"," \n"," ```python\n","[[0. 0. 0. ... 0. 0. 0.]\n"," [0. 0. 1. ... 0. 0. 0.]\n"," [0. 0. 0. ... 0. 0. 0.]\n"," ...\n"," [0. 0. 0. ... 0. 0. 0.]\n"," [0. 0. 0. ... 0. 0. 0.]\n"," [0. 0. 0. ... 0. 0. 0.]]\n","```\n","\n","This allows us to preserve the structural information but there are two major disadvantages here. We used character level representations in the CNN lessons because the number of characters is small. Suppose we wanted to one-hot encode each word instead. Now the vocabulary sizes quickly grows leading to large computes. And though we preserve the structure within the text, the actual representation for each token does not preserve any relationship with respect to other tokens.\n","\n","In this notebook, we're going to learn about embeddings and how they address all the shortcomings of the representation methods we've seen so far.\n","\n","\n","\n"]},{"cell_type":"markdown","metadata":{"id":"yN73ZCCnjezh","colab_type":"text"},"source":["* **Objective:**  Represent tokens in text that capture the intrinsic semantic relationships.\n","* **Advantages:** \n","    * Low-dimensionality while capturing relationships.\n","    * Interpretable token representations\n","* **Disadvantages:** None\n","* **Miscellaneous:** There are lot's of pretrained embeddings to choose from but you can also train your own from scratch."]},{"cell_type":"markdown","metadata":{"id":"MrDStrYbjsnW","colab_type":"text"},"source":["# Data"]},{"cell_type":"markdown","metadata":{"id":"nH_O4MZ294jk","colab_type":"text"},"source":["## Load data"]},{"cell_type":"markdown","metadata":{"id":"F47IiPgUupAk","colab_type":"text"},"source":["We can learn embeddings by creating our models in TensorFLow but instead, we're going to use a library that specializes in embeddings and topic modeling called [Gensim](https://radimrehurek.com/gensim/). "]},{"cell_type":"code","metadata":{"id":"NUuFGxRI8xxl","colab_type":"code","colab":{}},"source":["import urllib"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"lVfE26vR9O-y","colab_type":"code","colab":{}},"source":["DATA_FILE = 'harrypotter.txt'"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"LegtLIr-lxxZ","colab_type":"code","colab":{}},"source":["# Load data from GitHub to this notebook's local drive\n","url = \"https://raw.githubusercontent.com/madewithml/lessons/master/data/harrypotter.txt\"\n","response = urllib.request.urlopen(url)\n","html = response.read()\n","with open(DATA_FILE, 'wb') as fp:\n","    fp.write(html)"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"4ZDGEBa2-Ccf","colab_type":"text"},"source":["## Preprocess"]},{"cell_type":"code","metadata":{"id":"62qsAAZ5gG9M","colab_type":"code","outputId":"19d9a314-6b13-4ada-e42c-98837d5e9a61","executionInfo":{"status":"ok","timestamp":1584550699062,"user_tz":420,"elapsed":5299,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["# Use TensorFlow 2.x\n","%tensorflow_version 2.x"],"execution_count":4,"outputs":[{"output_type":"stream","text":["TensorFlow 2.x selected.\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"_pZljlaCgG6Y","colab_type":"code","outputId":"4d2314c0-0b20-4015-9a03-9ec242ec9bd8","executionInfo":{"status":"ok","timestamp":1584550701360,"user_tz":420,"elapsed":7579,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":68}},"source":["import nltk; nltk.download('punkt')\n","import tensorflow as tf\n","from tensorflow.keras.preprocessing.text import text_to_word_sequence\n","print(\"GPU Available: \", tf.config.list_physical_devices('GPU'))"],"execution_count":5,"outputs":[{"output_type":"stream","text":["[nltk_data] Downloading package punkt to /root/nltk_data...\n","[nltk_data]   Unzipping tokenizers/punkt.zip.\n","GPU Available:  [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"oektJd55gG1p","colab_type":"code","colab":{}},"source":["SEED = 1234"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"tqbnugiD-SW0","colab_type":"code","colab":{}},"source":["# Set seed for reproducibility\n","tf.random.set_seed(SEED)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"fdtG_em9YaLc","colab_type":"code","colab":{}},"source":["FILTERS = \"!\\\"'#$%&()*+,-./:;<=>?@[\\\\]^_`{|}~\"\n","LOWER = True"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"vF5D_nNjlx2d","colab_type":"code","outputId":"4429d5bf-4710-4d6f-c336-7be26feead7f","executionInfo":{"status":"ok","timestamp":1584550701362,"user_tz":420,"elapsed":7536,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["# Split text into sentences\n","tokenizer = nltk.data.load('tokenizers/punkt/english.pickle')\n","with open(DATA_FILE, encoding='cp1252') as fp:\n","    book = fp.read()\n","sentences = tokenizer.tokenize(book)\n","print (f\"{len(sentences)} sentences\")"],"execution_count":9,"outputs":[{"output_type":"stream","text":["15640 sentences\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"NsZz5jfMlx0d","colab_type":"code","outputId":"e48716b7-fe2c-455d-ffa4-229b5d55e947","executionInfo":{"status":"ok","timestamp":1584550701592,"user_tz":420,"elapsed":7736,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":51}},"source":["# Preprocess sentences\n","print (sentences[11])\n","sentences = [text_to_word_sequence(\n","    text=sentence, filters=FILTERS, \n","    lower=LOWER, split=' ') for sentence in sentences]\n","print (sentences[11])"],"execution_count":10,"outputs":[{"output_type":"stream","text":["Snape nodded, but did not elaborate.\n","['snape', 'nodded', 'but', 'did', 'not', 'elaborate']\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"yjIUhBwxkBHc","colab_type":"text"},"source":["# Learning embeddings"]},{"cell_type":"markdown","metadata":{"id":"rozFTf06ji1b","colab_type":"text"},"source":["The main idea of embeddings is to have fixed length representations for the tokens in a text regardless of the number of tokens in the vocabulary. So instead of each token representation having the shape [1 X V] where V is vocab size, each token now has the shape [1 X D] where D is the embedding size (usually 50, 100, 200, 300). The numbers in the representation will no longer be 0s and 1s but rather floats that represent that token in a D-dimensional latent space. If the embeddings really did capture the relationship between tokens, then we should be able to inspect this latent space and confirm known relationships (we'll do this soon).\n","\n","But how do we learn the embeddings the first place? The intuition behind embeddings is that the definition of a token depends on the token itself but on its context. There are several different ways of doing this:\n","\n","1. Given the word in the context, predict the target word (CBOW - continuous bag of words).\n","2. Given the target word, predict the context word (skip-gram).\n","3. Given a sequence of words, predict the next word (LM - language modeling).\n","\n","All of these approaches involve create data to train our model on. Every word in a sentence becomes the target word and the context words are determines by a window. In the image below (skip-gram), the window size is 2 (2 words to the left and right of the target word). We repeat this for every sentence in our corpus and this results in our training data for the unsupervised task. This in an unsupervised learning technique since we don't have official labels for contexts. The idea is that similar target words will appear with similar contexts and we can learn this relationship by repeatedly training our mode with (context, target) pairs.\n","\n","<div align=\"left\">\n","<img src=\"https://raw.githubusercontent.com/madewithml/images/master/02_Basics/08_Embeddings/skipgram.png\" width=\"600\">\n","</div>\n","\n","We can learn embeddings using any of these approaches above and some work better than others. You can inspect the learned embeddings but the best way to choose an approach is to empirically validate the performance on a supervised task."]},{"cell_type":"markdown","metadata":{"id":"No6c943C-P7o","colab_type":"text"},"source":["## Word2Vec"]},{"cell_type":"markdown","metadata":{"id":"VeszvcMOji4u","colab_type":"text"},"source":["When we have large vocabularies to learn embeddings for, things can get complex very quickly. Recall that the backpropagation with softmax updates both the correct and incorrect class weights. This becomes a massive computation for every backwas pass we do so a workaround is to use [negative sampling](http://mccormickml.com/2017/01/11/word2vec-tutorial-part-2-negative-sampling/) which only updates the correct class and a few arbitrary incorrect classes (negative_sampling=20). We're able to do this because of the large amount of training data where we'll see the same word as the target class multiple times.\n","\n"]},{"cell_type":"code","metadata":{"id":"TqKCr--k-f9e","colab_type":"code","colab":{}},"source":["import gensim\n","from gensim.models import KeyedVectors\n","from gensim.models import Word2Vec"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"ufU-9l_W-QKj","colab_type":"code","colab":{}},"source":["EMBEDDING_DIM = 100\n","WINDOW = 5\n","MIN_COUNT = 3 # Ignores all words with total frequency lower than this\n","SKIP_GRAM = 1 # 0 = CBOW\n","NEGATIVE_SAMPLING = 20"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"Ha3I2oSsmhJa","colab_type":"code","outputId":"fdb857f0-b642-487f-abc9-653630f9e25a","executionInfo":{"status":"ok","timestamp":1584550713317,"user_tz":420,"elapsed":19432,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["# Super fast because of optimized C code under the hood\n","w2v = Word2Vec(sentences=sentences, size=EMBEDDING_DIM, \n","               window=WINDOW, min_count=MIN_COUNT, \n","               sg=SKIP_GRAM, negative=NEGATIVE_SAMPLING)\n","print (w2v)"],"execution_count":13,"outputs":[{"output_type":"stream","text":["Word2Vec(vocab=4963, size=100, alpha=0.025)\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"Cl6oJv8jmhHE","colab_type":"code","outputId":"04b46ec1-bb3e-4dc3-ed64-e1d30b94ea93","executionInfo":{"status":"ok","timestamp":1584550713318,"user_tz":420,"elapsed":19413,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":374}},"source":["# Vector for each word\n","w2v.wv.get_vector(\"potter\")"],"execution_count":14,"outputs":[{"output_type":"execute_result","data":{"text/plain":["array([ 0.09083509,  0.03438687,  0.05106555, -0.14815712,  0.10591233,\n","        0.20836212,  0.24555065, -0.11337929, -0.15395121,  0.15665658,\n","       -0.20920357, -0.25585246,  0.00767487, -0.27146402, -0.38179845,\n","       -0.02202686,  0.02203755,  0.26650387, -0.20585173, -0.44206017,\n","        0.11651886,  0.08126067, -0.39437687,  0.23980044,  0.10549201,\n","       -0.33127534,  0.2485104 , -0.04941307,  0.10637701, -0.36734644,\n","        0.41268966, -0.17713442,  0.1050859 , -0.08241509,  0.04086861,\n","       -0.23368059,  0.31916997, -0.21991627,  0.24500914, -0.0773962 ,\n","       -0.0022594 ,  0.03968884,  0.05402524, -0.27722943,  0.0657296 ,\n","        0.09777772,  0.23136745,  0.11258315, -0.1276213 ,  0.01981706,\n","        0.22423954, -0.10857982,  0.08016241, -0.27785912,  0.3622481 ,\n","       -0.2329558 , -0.16779569, -0.03250792, -0.02298957, -0.13246015,\n","        0.08759305, -0.5524119 ,  0.42702344,  0.2022841 , -0.0753677 ,\n","        0.08785033,  0.4915804 ,  0.35160717,  0.26848456,  0.20791122,\n","       -0.2837811 ,  0.11716639,  0.09555206, -0.0038735 , -0.03485451,\n","       -0.86905736, -0.25433564, -0.36819017,  0.4031443 ,  0.23395498,\n","       -0.06089864, -0.10327473,  0.50203174, -0.08102859,  0.05355012,\n","       -0.08168992, -0.337022  , -0.16682921, -0.12174169,  0.53007054,\n","       -0.16049525, -0.13475178,  0.27101853, -0.0272716 , -0.04012604,\n","       -0.17241955, -0.03821262, -0.00168802, -0.15497006,  0.35935527],\n","      dtype=float32)"]},"metadata":{"tags":[]},"execution_count":14}]},{"cell_type":"code","metadata":{"id":"DyuLX9DTnLvM","colab_type":"code","outputId":"3a5e590c-f65c-49bd-966e-283bd124253f","executionInfo":{"status":"ok","timestamp":1584550713318,"user_tz":420,"elapsed":19394,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":156}},"source":["# Get nearest neighbors (excluding itself)\n","w2v.wv.most_similar(positive=\"scar\", topn=5)"],"execution_count":15,"outputs":[{"output_type":"stream","text":["/usr/local/lib/python3.6/dist-packages/gensim/matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n","  if np.issubdtype(vec.dtype, np.int):\n"],"name":"stderr"},{"output_type":"execute_result","data":{"text/plain":["[('pain', 0.949640154838562),\n"," ('forehead', 0.9127615094184875),\n"," ('burning', 0.9122476577758789),\n"," ('face', 0.9070782661437988),\n"," ('throat', 0.8985559344291687)]"]},"metadata":{"tags":[]},"execution_count":15}]},{"cell_type":"code","metadata":{"id":"YT7B0KRVTFew","colab_type":"code","outputId":"b64faca7-c609-492e-bb08-a7d0dddbd46a","executionInfo":{"status":"ok","timestamp":1584550713319,"user_tz":420,"elapsed":19371,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":71}},"source":["# Saving and loading\n","w2v.wv.save_word2vec_format('model.bin', binary=True)\n","w2v = KeyedVectors.load_word2vec_format('model.bin', binary=True)"],"execution_count":16,"outputs":[{"output_type":"stream","text":["/usr/local/lib/python3.6/dist-packages/smart_open/smart_open_lib.py:402: UserWarning: This function is deprecated, use smart_open.open instead. See the migration notes for details: https://github.com/RaRe-Technologies/smart_open/blob/master/README.rst#migrating-to-the-new-open-function\n","  'See the migration notes for details: %s' % _MIGRATION_NOTES_URL\n"],"name":"stderr"}]},{"cell_type":"markdown","metadata":{"id":"JZXVP5vfuiD5","colab_type":"text"},"source":["## FastText"]},{"cell_type":"markdown","metadata":{"id":"uvuoeWYMuqsa","colab_type":"text"},"source":["What happen's when a word doesn't exist in our vocabulary? We could assign an UNK token which is used for all OOV (out of vocabulary) words or we could use [FastText](https://radimrehurek.com/gensim/models/fasttext.html), which uses character-level n-grams to embed a word. This helps embed rare words, misspelled words, and also words that don't exist in our corpus but are similar to words in our corpus."]},{"cell_type":"code","metadata":{"id":"fVg3PBeD-kAa","colab_type":"code","colab":{}},"source":["from gensim.models import FastText"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"eTNW4Mfgrpo0","colab_type":"code","outputId":"7b6f9472-6ad5-4fd4-d151-80870c18f381","executionInfo":{"status":"ok","timestamp":1584550726329,"user_tz":420,"elapsed":32338,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["# Super fast because of optimized C code under the hood\n","ft = FastText(sentences=sentences, size=EMBEDDING_DIM, \n","              window=WINDOW, min_count=MIN_COUNT, \n","              sg=SKIP_GRAM, negative=NEGATIVE_SAMPLING)\n","print (ft)"],"execution_count":18,"outputs":[{"output_type":"stream","text":["FastText(vocab=4963, size=100, alpha=0.025)\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"LbA4vU5uxiw3","colab_type":"code","colab":{}},"source":["# This word doesn't exist so the word2vec model will error out\n","# w2v.wv.most_similar(positive=\"scarring\", topn=5)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"eRG30aE4sMjt","colab_type":"code","outputId":"fcb1ebf7-8d36-4ccd-fbb2-d28eac7969d4","executionInfo":{"status":"ok","timestamp":1584550726330,"user_tz":420,"elapsed":32317,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":156}},"source":["# FastText will use n-grams to embed an OOV word\n","ft.wv.most_similar(positive=\"scarring\", topn=5)"],"execution_count":20,"outputs":[{"output_type":"stream","text":["/usr/local/lib/python3.6/dist-packages/gensim/matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n","  if np.issubdtype(vec.dtype, np.int):\n"],"name":"stderr"},{"output_type":"execute_result","data":{"text/plain":["[('sparkling', 0.9855315685272217),\n"," ('spiraling', 0.9833263754844666),\n"," ('shimmering', 0.9828085899353027),\n"," ('rippling', 0.9827985167503357),\n"," ('trembling', 0.9822416305541992)]"]},"metadata":{"tags":[]},"execution_count":20}]},{"cell_type":"code","metadata":{"id":"7SE5fPMUnLyP","colab_type":"code","outputId":"3ad0f818-a265-487a-bf2a-15606734da66","executionInfo":{"status":"ok","timestamp":1584550726331,"user_tz":420,"elapsed":32287,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":71}},"source":["# Save and loading\n","ft.wv.save('model.bin')\n","ft = KeyedVectors.load('model.bin')"],"execution_count":21,"outputs":[{"output_type":"stream","text":["/usr/local/lib/python3.6/dist-packages/smart_open/smart_open_lib.py:402: UserWarning: This function is deprecated, use smart_open.open instead. See the migration notes for details: https://github.com/RaRe-Technologies/smart_open/blob/master/README.rst#migrating-to-the-new-open-function\n","  'See the migration notes for details: %s' % _MIGRATION_NOTES_URL\n"],"name":"stderr"}]},{"cell_type":"markdown","metadata":{"id":"67UmjtK0pF9X","colab_type":"text"},"source":["# Pretrained embeddings"]},{"cell_type":"markdown","metadata":{"id":"Xm1GPn4spF6x","colab_type":"text"},"source":["We can learn embeddings from scratch using one of the approaches above but we can also leverage pretrained embeddings that have been trained on millions of documents. Popular ones include Word2Vec (skip-gram) or GloVe (global word-word co-occurrence). We can validate that these embeddings captured meaningful semantic relationships by confirming them."]},{"cell_type":"code","metadata":{"id":"Hh42Mb4lLbuB","colab_type":"code","colab":{}},"source":["from gensim.scripts.glove2word2vec import glove2word2vec\n","from io import BytesIO\n","import matplotlib.pyplot as plt\n","import numpy as np\n","from sklearn.decomposition import PCA\n","from urllib.request import urlopen\n","from zipfile import ZipFile"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"pZIn8oAaBiZv","colab_type":"code","colab":{}},"source":["# Set seed for reproducibility\n","np.random.seed(SEED)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"m9gxHJA9M8hK","colab_type":"code","colab":{}},"source":["# Arguments\n","EMBEDDING_DIM = 100"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"ANfQHxGrMKTe","colab_type":"code","colab":{}},"source":["def plot_embeddings(words, embeddings, pca_results):\n","    for word in words:\n","        index = embeddings.index2word.index(word)\n","        plt.scatter(pca_results[index, 0], pca_results[index, 1])\n","        plt.annotate(word, xy=(pca_results[index, 0], pca_results[index, 1]))\n","    plt.show()"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"ZW9Qtkz3LfdY","colab_type":"code","outputId":"3d127209-370f-4202-c3d7-62cc160aad31","executionInfo":{"status":"ok","timestamp":1584551115734,"user_tz":420,"elapsed":421660,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":85}},"source":["# Unzip the file (may take ~3-5 minutes)\n","resp = urlopen('http://nlp.stanford.edu/data/glove.6B.zip')\n","zipfile = ZipFile(BytesIO(resp.read()))\n","zipfile.namelist()"],"execution_count":26,"outputs":[{"output_type":"execute_result","data":{"text/plain":["['glove.6B.50d.txt',\n"," 'glove.6B.100d.txt',\n"," 'glove.6B.200d.txt',\n"," 'glove.6B.300d.txt']"]},"metadata":{"tags":[]},"execution_count":26}]},{"cell_type":"code","metadata":{"id":"bWnVBrOaLjIC","colab_type":"code","outputId":"77d82016-3b1c-43e3-e24e-7cee672ac5d3","executionInfo":{"status":"ok","timestamp":1584551118950,"user_tz":420,"elapsed":424855,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["# Write embeddings to file\n","embeddings_file = 'glove.6B.{0}d.txt'.format(EMBEDDING_DIM)\n","zipfile.extract(embeddings_file)"],"execution_count":27,"outputs":[{"output_type":"execute_result","data":{"text/plain":["'/content/glove.6B.100d.txt'"]},"metadata":{"tags":[]},"execution_count":27}]},{"cell_type":"code","metadata":{"id":"qFLyIqIxrUIs","colab_type":"code","outputId":"3cc33b26-57e4-4cb2-e86b-26d37160b3f1","executionInfo":{"status":"ok","timestamp":1584551118950,"user_tz":420,"elapsed":424835,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":323}},"source":["# Preview of the GloVe embeddings file\n","with open(embeddings_file, 'r') as fp:\n","    line = next(fp)\n","    values = line.split()\n","    word = values[0]\n","    embedding = np.asarray(values[1:], dtype='float32')\n","    print (f\"word: {word}\")\n","    print (f\"embedding:\\n{embedding}\")\n","    print (f\"embedding dim: {len(embedding)}\")"],"execution_count":28,"outputs":[{"output_type":"stream","text":["word: the\n","embedding:\n","[-0.038194 -0.24487   0.72812  -0.39961   0.083172  0.043953 -0.39141\n","  0.3344   -0.57545   0.087459  0.28787  -0.06731   0.30906  -0.26384\n"," -0.13231  -0.20757   0.33395  -0.33848  -0.31743  -0.48336   0.1464\n"," -0.37304   0.34577   0.052041  0.44946  -0.46971   0.02628  -0.54155\n"," -0.15518  -0.14107  -0.039722  0.28277   0.14393   0.23464  -0.31021\n","  0.086173  0.20397   0.52624   0.17164  -0.082378 -0.71787  -0.41531\n","  0.20335  -0.12763   0.41367   0.55187   0.57908  -0.33477  -0.36559\n"," -0.54857  -0.062892  0.26584   0.30205   0.99775  -0.80481  -3.0243\n","  0.01254  -0.36942   2.2167    0.72201  -0.24978   0.92136   0.034514\n","  0.46745   1.1079   -0.19358  -0.074575  0.23353  -0.052062 -0.22044\n","  0.057162 -0.15806  -0.30798  -0.41625   0.37972   0.15006  -0.53212\n"," -0.2055   -1.2526    0.071624  0.70565   0.49744  -0.42063   0.26148\n"," -1.538    -0.30223  -0.073438 -0.28312   0.37104  -0.25217   0.016215\n"," -0.017099 -0.38984   0.87424  -0.72569  -0.51058  -0.52028  -0.1459\n","  0.8278    0.27062 ]\n","embedding dim: 100\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"9eD5doqFLjFY","colab_type":"code","outputId":"06a8d0ec-a862-4495-8f3c-140e57d9d33b","executionInfo":{"status":"ok","timestamp":1584551120204,"user_tz":420,"elapsed":426068,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":88}},"source":["# Save GloVe embeddings to local directory in word2vec format\n","word2vec_output_file = '{0}.word2vec'.format(embeddings_file)\n","glove2word2vec(embeddings_file, word2vec_output_file)"],"execution_count":29,"outputs":[{"output_type":"stream","text":["/usr/local/lib/python3.6/dist-packages/smart_open/smart_open_lib.py:402: UserWarning: This function is deprecated, use smart_open.open instead. See the migration notes for details: https://github.com/RaRe-Technologies/smart_open/blob/master/README.rst#migrating-to-the-new-open-function\n","  'See the migration notes for details: %s' % _MIGRATION_NOTES_URL\n"],"name":"stderr"},{"output_type":"execute_result","data":{"text/plain":["(400000, 100)"]},"metadata":{"tags":[]},"execution_count":29}]},{"cell_type":"code","metadata":{"id":"To4sx_1iMCX0","colab_type":"code","outputId":"0a9afc39-967d-4a9f-9f36-e6c2a1861ba4","executionInfo":{"status":"ok","timestamp":1584551155794,"user_tz":420,"elapsed":461634,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":71}},"source":["# Load embeddings (may take a minute)\n","glove = KeyedVectors.load_word2vec_format(word2vec_output_file, binary=False)"],"execution_count":30,"outputs":[{"output_type":"stream","text":["/usr/local/lib/python3.6/dist-packages/smart_open/smart_open_lib.py:402: UserWarning: This function is deprecated, use smart_open.open instead. See the migration notes for details: https://github.com/RaRe-Technologies/smart_open/blob/master/README.rst#migrating-to-the-new-open-function\n","  'See the migration notes for details: %s' % _MIGRATION_NOTES_URL\n"],"name":"stderr"}]},{"cell_type":"code","metadata":{"id":"UEhBhvgHMEH9","colab_type":"code","outputId":"55aff262-b918-4f7f-f2c5-c28c6694fe3e","executionInfo":{"status":"ok","timestamp":1584551155796,"user_tz":420,"elapsed":461611,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":156}},"source":["# (king - man) + woman = ?\n","glove.most_similar(positive=['woman', 'king'], negative=['man'], topn=5)"],"execution_count":31,"outputs":[{"output_type":"stream","text":["/usr/local/lib/python3.6/dist-packages/gensim/matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n","  if np.issubdtype(vec.dtype, np.int):\n"],"name":"stderr"},{"output_type":"execute_result","data":{"text/plain":["[('queen', 0.7698541283607483),\n"," ('monarch', 0.6843380928039551),\n"," ('throne', 0.6755735874176025),\n"," ('daughter', 0.6594556570053101),\n"," ('princess', 0.6520534753799438)]"]},"metadata":{"tags":[]},"execution_count":31}]},{"cell_type":"code","metadata":{"id":"xR94AICkMEFV","colab_type":"code","outputId":"558efa58-3085-4cf5-b041-3fe62614cc59","executionInfo":{"status":"ok","timestamp":1584551155796,"user_tz":420,"elapsed":461584,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":190}},"source":["# Get nearest neighbors (exlcusing itself)\n","glove.wv.most_similar(positive=\"goku\", topn=5)"],"execution_count":32,"outputs":[{"output_type":"stream","text":["/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:1: DeprecationWarning: Call to deprecated `wv` (Attribute will be removed in 4.0.0, use self instead).\n","  \"\"\"Entry point for launching an IPython kernel.\n","/usr/local/lib/python3.6/dist-packages/gensim/matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n","  if np.issubdtype(vec.dtype, np.int):\n"],"name":"stderr"},{"output_type":"execute_result","data":{"text/plain":["[('gohan', 0.7246542572975159),\n"," ('bulma', 0.6497020125389099),\n"," ('raistlin', 0.6443604230880737),\n"," ('skaar', 0.6316742897033691),\n"," ('guybrush', 0.6231324672698975)]"]},"metadata":{"tags":[]},"execution_count":32}]},{"cell_type":"code","metadata":{"id":"gseqjBmzMECq","colab_type":"code","outputId":"3ea3efef-c735-4b50-fd37-5c1d70554576","executionInfo":{"status":"ok","timestamp":1584551158749,"user_tz":420,"elapsed":464520,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":51}},"source":["# Reduce dimensionality for plotting\n","X = glove[glove.wv.vocab]\n","pca = PCA(n_components=2)\n","pca_results = pca.fit_transform(X)"],"execution_count":33,"outputs":[{"output_type":"stream","text":["/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:1: DeprecationWarning: Call to deprecated `wv` (Attribute will be removed in 4.0.0, use self instead).\n","  \"\"\"Entry point for launching an IPython kernel.\n"],"name":"stderr"}]},{"cell_type":"code","metadata":{"id":"LFQWGyncMHgK","colab_type":"code","outputId":"94aeb639-72db-47a8-a08a-7d29a3ec53ca","executionInfo":{"status":"ok","timestamp":1584551158751,"user_tz":420,"elapsed":464505,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":265}},"source":["# Visualize\n","plot_embeddings(words=[\"king\", \"queen\", \"man\", \"woman\"], \n","                embeddings=glove, \n","                pca_results=pca_results)"],"execution_count":34,"outputs":[{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAXsAAAD4CAYAAAANbUbJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjAsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8GearUAAAV1ElEQVR4nO3df5BV9Znn8fdDg90oDuyGxhDQgLXg\nD2ykm8Yag8QWs4KiMFNJHCmyGyeKqcSJxkQ0Zo2yWkllgrX+yGY0OKFQU6ImKgWoI/5ARR0HGkUm\noAiLPSvoCLqmIwKRxu/+0W1Pg0Df7r59L7fP+1XVVfd8z/ec8zx1qY/Hc869N1JKSJJ6tl7FLkCS\n1P0Me0nKAMNekjLAsJekDDDsJSkDehfrwAMHDkzDhg0r1uElqSStWrXqvZRSZUe3K1rYDxs2jPr6\n+mIdXpJKUkT8W2e28zKOJGWAYV/iGhoaOOmkk/Yaq6+v57LLLitSRZIORUW7jKPuU1tbS21tbbHL\nkHQI8cy+B9m0aRPV1dXMmTOHc889F4DZs2fzrW99i7q6Oo499lhuu+221vk33ngjxx13HKeddhrT\np0/npptuKlbpkrqZZ/Y9xPr167nggguYP38+H3zwAc8++2zrutdff51ly5bx4Ycfctxxx/Gd73yH\n1atX8+CDD/Lqq6+ye/duampqGDt2bBE7kNSdDPsStPCVLcx5fD1v/3En/zk1svmdd5k2bRoPPfQQ\nJ554Is8888xe86dMmUJ5eTnl5eUMGjSId999lxdeeIFp06ZRUVFBRUUF5513XnGakVQQXsYpMQtf\n2cI1D/0rW/64kwS8+6dd7KCciv90FM8///x+tykvL299XVZWRlNTU4GqlXSoMOxLzJzH17Nz9569\nB3uVUXH2Vdx9993ce++9Oe1n/PjxLF68mF27drF9+3aWLFnSDdVKOlQY9iXm7T/u3O/4uztgyZIl\n3HzzzfzpT39qdz/jxo1j6tSpjB49mrPPPpuqqir69++f73IlHSKiWD9eUltbm/wEbceN//nTbNlP\n4A8Z0JcXfjSxQ/vavn07/fr1Y8eOHXz5y19m7ty51NTU5KtUSd0gIlallDr8bHW7Z/YRMS8itkbE\nH9qZNy4imiLiax0tQrmbNek4+vYp22usb58yZk06rsP7uuSSSxgzZgw1NTV89atfNeilHiyXp3Hm\nA/8buPtAEyKiDPh7YGl+ytKB/FX1EIDWp3G+MKAvsyYd1zreEble35dU+toN+5TScxExrJ1p3wMe\nBMbloSa146+qh3Qq3CVlV5dv0EbEEOCvgdtzmHtJRNRHRP22bdu6emhJUo7y8TTOLcDVKaVP2puY\nUpqbUqpNKdVWVnb465glSZ2Uj0/Q1gL3RQTAQOCciGhKKS3Mw74lSXnQ5bBPKQ3/9HVEzAeWGPSS\ndGhpN+wjYgFQBwyMiM3A9UAfgJTSHd1anSQpL3J5Gmd6rjtLKV3YpWokSd3Cr0uQpAww7CUpAwx7\nScoAw16SMsCwl6QMMOwlKQMMe0nKAMNekjLAsJekDDDsJSkDDHtJygDDXpIywLCXpAww7CUpAwx7\nScoAw16SMsCwl6QMMOwlKQMMe0nKAMNekjLAsJekDDDsJSkDDHtJygDDXpIywLCXpAww7CUpAwx7\nScoAw16SMsCwl6QMMOwlKQMMe0nKAMNekjLAsJekDDDsJSkDDHtJyoB2wz4i5kXE1oj4wwHWz4iI\nNRHxrxHxYkScnP8yJUldkcuZ/Xxg8kHWvwmcnlKqAm4E5uahLklSHvVub0JK6bmIGHaQ9S+2WXwJ\nGNr1siRJ+ZTva/YXAY8daGVEXBIR9RFRv23btjwfWpJ0IHkL+4g4g+awv/pAc1JKc1NKtSml2srK\nynwdWpLUjnYv4+QiIkYD/wicnVJ6Px/7lCTlT5fP7CPiGOAh4L+llN7oekmSpHxr98w+IhYAdcDA\niNgMXA/0AUgp3QFcB3wO+IeIAGhKKdV2V8GSpI7L5Wmc6e2svxi4OG8VSZLyzk/QSlIGGPaSlAGG\nvSRlgGEvSRlg2EtSBhj2kpQBhr0kZYBhL0kZYNhLUgYY9pKUAYa9JGWAYS9JGWDYS1IGGPaSlAGG\nvSRlgGEvSRlg2EtSBhj2kpQBhr0kFUFDQwPHH388F154ISNHjmTGjBk8+eSTjB8/nhEjRrBixQpW\nrFjBqaeeSnV1NV/60pdYv349ABFxYUQ8FBH/FBEbIuIX7R2v3d+glSR1j40bN/K73/2OefPmMW7c\nOO69916ef/55Fi1axM9+9jPuvvtuli9fTu/evXnyySf58Y9/3HbzMUA18GdgfUT8MqX01oGOZdhL\nUpEMHz6cqqoqAEaNGsWZZ55JRFBVVUVDQwONjY1885vfZMOGDUQEu3fvbrv5UymlRoCIWAd8ETDs\nJanYHtn0CLe+fCv//tG/0/+j/nwcH7eu69WrF+Xl5a2vm5qa+MlPfsIZZ5zBww8/TENDA3V1dW13\n9+c2r/fQTp57zV6SCuCRTY8w+8XZvPPROyQSW3dsZeuOrTyy6ZEDbtPY2MiQIUMAmD9/fpeOb9hL\nUgHc+vKt7Nqza6+xROLWl2894DZXXXUV11xzDdXV1TQ1NXXp+JFS6tIOOqu2tjbV19cX5diSVGij\n7xpN4rN5GwRrvrkm5/1ExKqUUm1Hj++ZvSQVwOeP+HyHxvPNsJekAri85nIqyir2Gqsoq+DymssL\ncnyfxpGkAphy7BSA1qdxPn/E57m85vLW8e5m2EtSgUw5dkrBwn1fXsaRpAww7CUpAwx7ScoAw16S\nMsCwl6QMMOwlKQPaDfuImBcRWyPiDwdYHxFxW0RsjIg1EVGT/zIlSV2Ry5n9fGDyQdafDYxo+bsE\nuL3rZUmS8qndsE8pPQf8v4NMmQbcnZq9BAyIiMH5KlCS1HX5uGY/hL1/HWVzy9hnRMQlEVEfEfXb\ntm3Lw6ElSbko6A3alNLclFJtSqm2srKykIeWpEzLR9hvAY5uszy0ZUySdIjIR9gvAv57y1M5fwk0\nppTeycN+JUl50u63XkbEAqAOGBgRm4HrgT4AKaU7gEeBc4CNwA7gb7urWElS57Qb9iml6e2sT8Cl\neatIkpR3foJWkjLAsJekDDDsJSkDDHtJygDDXpIywLCXpAww7CUpAwx7ScoAw16SMsCwl6QMMOwl\nKQMMe0nKAMNekjLAsJekDDDsJSkDDHtJygDDXpIywLCXpAww7CUpAwx7ScoAw16SMsCwl6QMMOwl\nKQMMe0nKAMNekjLAsJekDDDsJSkDDHtJygDDXpIywLCXpAww7CUpAwx7ScoAw16SMsCwl6QMMOwl\nKQNyCvuImBwR6yNiY0T8aD/rj4mIZRHxSkSsiYhz8l+qJKmz2g37iCgDfgWcDZwITI+IE/eZdi3w\nQEqpGrgA+Id8FypJ6rxczuxPATamlDallD4G7gOm7TMnAX/R8ro/8Hb+SpQkdVUuYT8EeKvN8uaW\nsbZmA9+IiM3Ao8D39rejiLgkIuojon7btm2dKFeS1Bn5ukE7HZifUhoKnAPcExGf2XdKaW5KqTal\nVFtZWZmnQ0uS2pNL2G8Bjm6zPLRlrK2LgAcAUkr/DFQAA/NRoCSp63IJ+5XAiIgYHhGH0XwDdtE+\nc/4vcCZARJxAc9h7nUaSDhHthn1KqQn4O+Bx4DWan7pZGxE3RMTUlmk/BGZGxKvAAuDClFLqrqIl\nSR3TO5dJKaVHab7x2nbsujav1wHj81uaJClf/AStJGWAYS9JGWDYS1IGGPaSlAGGvSRlgGEvSRlg\n2EtSBhj2kpQBhr0kZYBhL0kZYNhLUgYY9pKUAYa9JGWAYS9JGWDYSwU0Z84cbrvtNgCuuOIKJk6c\nCMDTTz/NjBkzWLBgAVVVVZx00klcffXVrdv169ePWbNmMWrUKL7yla+wYsUK6urqOPbYY1m0qPm3\nhBoaGpgwYQI1NTXU1NTw4osvAvDMM89QV1fH1772NY4//nhmzJiBPzeRPYa9VEATJkxg+fLlANTX\n17N9+3Z2797N8uXLGTlyJFdffTVPP/00q1evZuXKlSxcuBCAjz76iIkTJ7J27VqOPPJIrr32Wp54\n4gkefvhhrruu+aclBg0axBNPPMHLL7/M/fffz2WXXdZ63FdeeYVbbrmFdevWsWnTJl544YXCN6+i\nMuylAmhcvJgNE8/k8Av/lpeWLOGt+++nvLycU089lfr6epYvX86AAQOoq6ujsrKS3r17M2PGDJ57\n7jkADjvsMCZPngxAVVUVp59+On369KGqqoqGhgYAdu/ezcyZM6mqquLrX/8669ataz3+KaecwtCh\nQ+nVqxdjxoxp3UbZYdhL3axx8WLe+cl1NL39Nn2AIb16cccPfkjNwIFMmDCBZcuWsXHjRoYNG3bA\nffTp04eIAKBXr16Ul5e3vm5qagLg5ptv5qijjuLVV1+lvr6ejz/+uHX7T+cDlJWVtW6j7DDspW62\n9eZbSLt2tS6P7duXeVvf5cTX1zNhwgTuuOMOqqurOeWUU3j22Wd577332LNnDwsWLOD000/P+TiN\njY0MHjyYXr16cc8997Bnz57uaEclyrCXulnTO+/stTy27+G819RE1c6dHHXUUVRUVDBhwgQGDx7M\nz3/+c8444wxOPvlkxo4dy7Rp03I+zne/+13uuusuTj75ZF5//XWOOOKIfLeiEhbFuitfW1ub6uvr\ni3JsqZA2TDyTprff/sx47y98gRFPP1WEilTKImJVSqm2o9t5Zi91s0FXfJ+oqNhrLCoqGHTF94tU\nkbKod7ELkHq6/uedBzRfu2965x16Dx7MoCu+3zouFYJhLxVA//POM9xVVF7GkaQMMOwlKQMMe0nK\nAMNekjLAsJekDDDsJSkDDHtJygDDXpIywLCXpAww7CUpAwx7ScqAnMI+IiZHxPqI2BgRPzrAnPMj\nYl1ErI2Ie/NbpiSpK9r9IrSIKAN+BfxXYDOwMiIWpZTWtZkzArgGGJ9S+iAiBnVXwZKkjsvlzP4U\nYGNKaVNK6WPgPmDfn8+ZCfwqpfQBQEppa37LlCR1RS5hPwR4q83y5paxtkYCIyPihYh4KSIm729H\nEXFJRNRHRP22bds6V7EkqcPydYO2NzACqAOmA3dGxIB9J6WU5qaUalNKtZWVlXk6tCSpPbmE/Rbg\n6DbLQ1vG2toMLEop7U4pvQm8QXP4S5IOAbmE/UpgREQMj4jDgAuARfvMWUjzWT0RMZDmyzqb8lin\nJKkL2g37lFIT8HfA48BrwAMppbURcUNETG2Z9jjwfkSsA5YBs1JK73dX0ZKkjomUUlEOXFtbm+rr\n64tybEkqVRGxKqVU29Ht/AStJGWAYS9JGWDYS1IGGPaSlAGGvSRlgGEvSRlQ0mH/05/+lJEjR3La\naacxffp0brrpJurq6vj0kc733nuPYcOGAbBnzx5mzZrFuHHjGD16NL/+9a9b9zNnzpzW8euvvx6A\nhoYGTjjhBGbOnMmoUaM466yz2LlzZ8F7lKR8KNmwX7VqFffddx+rV6/m0UcfZeXKlQed/5vf/Ib+\n/fuzcuVKVq5cyZ133smbb77J0qVL2bBhAytWrGD16tWsWrWK5557DoANGzZw6aWXsnbtWgYMGMCD\nDz5YiNYkKe/a/T77Q8qaB+CpG6BxM8tX9+WvvzSeww8/HICpU6cedNOlS5eyZs0afv/73wPQ2NjI\nhg0bWLp0KUuXLqW6uhqA7du3s2HDBo455hiGDx/OmDFjABg7diwNDQ3d15skdaPSCfs1D8Diy2B3\ny6WUXR/AG//UPD76/NZpvXv35pNPPmmesmtX63hKiV/+8pdMmjRpr90+/vjjXHPNNXz729/ea7yh\noYHy8vLW5bKyMi/jSCpZpXMZ56kb/iPogS9/sTcL1+1k52Oz+fDDD1m8eDEAw4YNY9WqVQCtZ/EA\nkyZN4vbbb2f37t0AvPHGG3z00UdMmjSJefPmsX37dgC2bNnC1q3+9oqknqV0zuwbN++1WDO4jL8Z\n1YeTf7GeQYvPZty4cQBceeWVnH/++cydO5cpU6a0zr/44otpaGigpqaGlBKVlZUsXLiQs846i9de\ne41TTz0VgH79+vHb3/6WsrKywvUmSd2sdL4I7eaToPGtz473Pxqu+AOzZ8+mX79+XHnllfkrUpIO\nMT3/i9DOvA769N17rE/f5nFJ0kGVzmWcT2/CtjyNQ/+hzUHfMj579uzi1SZJh7jSCXtoDvY2T95I\nknJTOpdxJEmdZthLUgYY9pKUAYa9JGWAYS9JGVC0D1VFxDbg37pp9wOB97pp38XWU3vrqX1Bz+2t\np/YFh3ZvX0wpVXZ0o6KFfXeKiPrOfMKsFPTU3npqX9Bze+upfUHP7M3LOJKUAYa9JGVATw37ucUu\noBv11N56al/Qc3vrqX1BD+ytR16zlyTtraee2UuS2jDsJSkDSjbsI6IiIlZExKsRsTYi/ucB5p0f\nEeta5txb6Do7I5feIuKYiFgWEa9ExJqIOKcYtXZGRJS11L1kP+vKI+L+iNgYEf8SEcMKX2HntNPX\nD1r+Ha6JiKci4ovFqLGzDtZbmzlfjYgUESXzyGJ7fZVifhxIaX3F8d7+DExMKW2PiD7A8xHxWErp\npU8nRMQI4BpgfErpg4gYVKxiO6jd3oBrgQdSSrdHxInAo8CwItTaGZcDrwF/sZ91FwEfpJT+S0Rc\nAPw98DeFLK4LDtbXK0BtSmlHRHwH+AWl0xccvDci4siWOf9SyKLy4IB9lXB+7FfJntmnZttbFvu0\n/O17t3km8KuU0gct25TEL4nn2FviP/6B9gfeLlB5XRIRQ4EpwD8eYMo04K6W178HzoyIKERtXdFe\nXymlZSmlHS2LLwFDC1VbV+XwngHcSPN/mHcVpKg8yKGvksyPAynZsIfW/wVbDWwFnkgp7XtWMRIY\nGREvRMRLETG58FV2Tg69zQa+ERGbaT6r/16BS+ysW4CrgE8OsH4I8BZASqkJaAQ+V5jSuqS9vtq6\nCHise8vJq4P2FhE1wNEppUcKWlXXtfeelWx+7E9Jh31KaU9KaQzNZ0mnRMRJ+0zpDYwA6oDpwJ0R\nMaCwVXZODr1NB+anlIYC5wD3RMQh/X5GxLnA1pTSqmLXkk8d6SsivgHUAnO6vbA8aK+3ln9z/wv4\nYUEL66Ic37OSzY/9OaTDIVcppT8Cy4B9/8u7GViUUtqdUnoTeIPmN69kHKS3i4AHWub8M1BB85c3\nHcrGA1MjogG4D5gYEb/dZ84W4GiAiOhN8yWq9wtZZCfk0hcR8RXgfwBTU0p/LmyJndZeb0cCJwHP\ntMz5S2BRCdykzeU9K/n82EtKqST/gEpgQMvrvsBy4Nx95kwG7mp5PZDmywOfK3bteertMeDCltcn\n0HzNPopdewd6rAOW7Gf8UuCOltcX0HwTuuj15qGvauD/ACOKXWO+e9tnzjM034guer15eM9KMj8O\n9FfKZ/aDgWURsQZYSfN17SURcUNETG2Z8zjwfkSso/nseFZK6VA/S4TcevshMDMiXgUW0Bz8Jflx\n6H36+g3wuYjYCPwA+FHxKuuaffqaA/QDfhcRqyNiURFL67J9eusxekh+7JdflyBJGVDKZ/aSpBwZ\n9pKUAYa9JGWAYS9JGWDYS1IGGPaSlAGGvSRlwP8HYu41jSQJwzYAAAAASUVORK5CYII=\n","text/plain":["<Figure size 432x288 with 1 Axes>"]},"metadata":{"tags":[]}}]},{"cell_type":"code","metadata":{"id":"MzrZ2_RBMHdn","colab_type":"code","outputId":"d1827acc-0867-4a8f-a0eb-a96b9f6dd822","executionInfo":{"status":"ok","timestamp":1584551158751,"user_tz":420,"elapsed":464486,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":156}},"source":["# Bias in embeddings\n","glove.most_similar(positive=['woman', 'doctor'], negative=['man'], topn=5)"],"execution_count":35,"outputs":[{"output_type":"stream","text":["/usr/local/lib/python3.6/dist-packages/gensim/matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n","  if np.issubdtype(vec.dtype, np.int):\n"],"name":"stderr"},{"output_type":"execute_result","data":{"text/plain":["[('nurse', 0.7735227346420288),\n"," ('physician', 0.7189429998397827),\n"," ('doctors', 0.6824328303337097),\n"," ('patient', 0.6750682592391968),\n"," ('dentist', 0.6726033687591553)]"]},"metadata":{"tags":[]},"execution_count":35}]},{"cell_type":"markdown","metadata":{"id":"xF1olr2citGG","colab_type":"text"},"source":["# Data"]},{"cell_type":"markdown","metadata":{"id":"c69z9wpJ56nE","colab_type":"text"},"source":["## Load data"]},{"cell_type":"markdown","metadata":{"id":"2V_nEp5G58M0","colab_type":"text"},"source":["We will download the [AG News dataset](http://www.di.unipi.it/~gulli/AG_corpus_of_news_articles.html), which consists of 120000 text samples from 4 unique classes ('Business', 'Sci/Tech', 'Sports', 'World')"]},{"cell_type":"code","metadata":{"id":"y3qKSoEe57na","colab_type":"code","colab":{}},"source":["import pandas as pd\n","import re\n","import urllib"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"N18Rcha5CL_t","colab_type":"code","colab":{}},"source":["DATA_FILE = 'news.csv'\n","INPUT_FEATURE = 'title'\n","OUTPUT_FEATURE = 'category'"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"cGQo98566GIV","colab_type":"code","colab":{}},"source":["# Upload data from GitHub to notebook's local drive\n","url = \"https://raw.githubusercontent.com/madewithml/lessons/master/data/news.csv\"\n","response = urllib.request.urlopen(url)\n","html = response.read()\n","with open(DATA_FILE, 'wb') as fp:\n","    fp.write(html)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"dG_Oltib6G-9","colab_type":"code","outputId":"a7b38751-d973-48c9-f965-20b61ec54152","executionInfo":{"status":"ok","timestamp":1584551159502,"user_tz":420,"elapsed":465207,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":204}},"source":["# Load data\n","df = pd.read_csv(DATA_FILE, header=0)\n","X = df[INPUT_FEATURE].values\n","y = df[OUTPUT_FEATURE].values\n","df.head(5)"],"execution_count":39,"outputs":[{"output_type":"execute_result","data":{"text/html":["<div>\n","<style scoped>\n","    .dataframe tbody tr th:only-of-type {\n","        vertical-align: middle;\n","    }\n","\n","    .dataframe tbody tr th {\n","        vertical-align: top;\n","    }\n","\n","    .dataframe thead th {\n","        text-align: right;\n","    }\n","</style>\n","<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>title</th>\n","      <th>category</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>Wall St. Bears Claw Back Into the Black (Reuters)</td>\n","      <td>Business</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>Carlyle Looks Toward Commercial Aerospace (Reu...</td>\n","      <td>Business</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>Oil and Economy Cloud Stocks' Outlook (Reuters)</td>\n","      <td>Business</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>Iraq Halts Oil Exports from Main Southern Pipe...</td>\n","      <td>Business</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>Oil prices soar to all-time record, posing new...</td>\n","      <td>Business</td>\n","    </tr>\n","  </tbody>\n","</table>\n","</div>"],"text/plain":["                                               title  category\n","0  Wall St. Bears Claw Back Into the Black (Reuters)  Business\n","1  Carlyle Looks Toward Commercial Aerospace (Reu...  Business\n","2    Oil and Economy Cloud Stocks' Outlook (Reuters)  Business\n","3  Iraq Halts Oil Exports from Main Southern Pipe...  Business\n","4  Oil prices soar to all-time record, posing new...  Business"]},"metadata":{"tags":[]},"execution_count":39}]},{"cell_type":"markdown","metadata":{"id":"hxo6RKCQ71dl","colab_type":"text"},"source":["## Split data"]},{"cell_type":"code","metadata":{"id":"eS6kCcfY6IHE","colab_type":"code","colab":{}},"source":["import collections\n","from sklearn.model_selection import train_test_split"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"55M6iPpZCRWH","colab_type":"code","colab":{}},"source":["TRAIN_SIZE = 0.7\n","VAL_SIZE = 0.15\n","TEST_SIZE = 0.15\n","SHUFFLE = True"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"-ZFVitqVWY4J","colab_type":"code","colab":{}},"source":["def train_val_test_split(X, y, val_size, test_size, shuffle):\n","    \"\"\"Split data into train/val/test datasets.\n","    \"\"\"\n","    X_train, X_test, y_train, y_test = train_test_split(\n","        X, y, test_size=test_size, stratify=y, shuffle=shuffle)\n","    X_train, X_val, y_train, y_val = train_test_split(\n","        X_train, y_train, test_size=val_size, stratify=y_train, shuffle=shuffle)\n","    return X_train, X_val, X_test, y_train, y_val, y_test"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"kqiQd2j_76gP","colab_type":"code","outputId":"81586394-a8b3-4718-b3cb-db12e67853b1","executionInfo":{"status":"ok","timestamp":1584551159683,"user_tz":420,"elapsed":465360,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":102}},"source":["# Create data splits\n","X_train, X_val, X_test, y_train, y_val, y_test = train_val_test_split(\n","    X=X, y=y, val_size=VAL_SIZE, test_size=TEST_SIZE, shuffle=SHUFFLE)\n","class_counts = dict(collections.Counter(y))\n","print (f\"X_train: {X_train.shape}, y_train: {y_train.shape}\")\n","print (f\"X_val: {X_val.shape}, y_val: {y_val.shape}\")\n","print (f\"X_test: {X_test.shape}, y_test: {y_test.shape}\")\n","print (f\"Sample point: {X_train[0]} → {y_train[0]}\")\n","print (f\"Classes: {class_counts}\")"],"execution_count":43,"outputs":[{"output_type":"stream","text":["X_train: (86700,), y_train: (86700,)\n","X_val: (15300,), y_val: (15300,)\n","X_test: (18000,), y_test: (18000,)\n","Sample point: Deep Impact Space Probe Aims to Slam Into Comet (Reuters) → Sci/Tech\n","Classes: {'Business': 30000, 'Sci/Tech': 30000, 'Sports': 30000, 'World': 30000}\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"dIfmW7vJ8Jx1","colab_type":"text"},"source":["## Tokenizer"]},{"cell_type":"markdown","metadata":{"id":"JP4VCO0LAJUt","colab_type":"text"},"source":["Unlike the previous notebook, we will be processing our text at a word-level (as opposed to character-level)."]},{"cell_type":"code","metadata":{"id":"DHPAxkKR7736","colab_type":"code","colab":{}},"source":["from tensorflow.keras.preprocessing.text import Tokenizer\n","from tensorflow.keras.utils import to_categorical"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"rDOj4s1ECa8q","colab_type":"code","colab":{}},"source":["FILTERS = \"!\\\"'#$%&()*+,-./:;<=>?@[\\\\]^_`{|}~\"\n","LOWER = True\n","CHAR_LEVEL = False"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"dMg5QhVybVfL","colab_type":"code","colab":{}},"source":["def decode(indices, tokenizer):\n","    \"\"\"Decode a list of indices into string.\"\"\"\n","    return \" \".join([tokenizer.index_word[index] for index in indices])"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"WcscM_vL8KvP","colab_type":"code","colab":{}},"source":["# Input vectorizer\n","X_tokenizer = Tokenizer(\n","    filters=FILTERS, lower=LOWER, char_level=CHAR_LEVEL, oov_token='<UNK>')"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"xV2JgpOA8PwO","colab_type":"code","outputId":"067cb0d8-a514-464d-868b-12031175b6b7","executionInfo":{"status":"ok","timestamp":1584551161792,"user_tz":420,"elapsed":467433,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["# Fit only on train data\n","X_tokenizer.fit_on_texts(X_train)\n","vocab_size = len(X_tokenizer.word_index) + 1\n","print (f\"# tokens: {vocab_size}\")"],"execution_count":48,"outputs":[{"output_type":"stream","text":["# tokens: 29795\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"ybb-YZSz8Qno","colab_type":"code","outputId":"6541a5ad-a45a-4859-b0c5-e4786974ae8f","executionInfo":{"status":"ok","timestamp":1584551162742,"user_tz":420,"elapsed":468360,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":68}},"source":["# Convert text to sequence of tokens\n","original_text = X_train[0]\n","X_train = np.array(X_tokenizer.texts_to_sequences(X_train))\n","X_val = np.array(X_tokenizer.texts_to_sequences(X_val))\n","X_test = np.array(X_tokenizer.texts_to_sequences(X_test))\n","preprocessed_text = decode(X_train[0], X_tokenizer)\n","print (f\"{original_text} \\n\\t→ {preprocessed_text} \\n\\t→ {X_train[0]}\")"],"execution_count":49,"outputs":[{"output_type":"stream","text":["Deep Impact Space Probe Aims to Slam Into Comet (Reuters) \n","\t→ deep impact space probe aims to slam into comet reuters \n","\t→ [2072, 1544, 92, 197, 563, 2, 2194, 69, 4804, 16]\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"ORGuhjCf8TKh","colab_type":"text"},"source":["## LabelEncoder"]},{"cell_type":"code","metadata":{"id":"7aBBgzkW8Rxv","colab_type":"code","colab":{}},"source":["from sklearn.preprocessing import LabelEncoder"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"ckM_MnQi8UTH","colab_type":"code","colab":{}},"source":["# Output vectorizer\n","y_tokenizer = LabelEncoder()"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"0-FkxqCT8WUk","colab_type":"code","outputId":"e95f41ab-0e7a-47b2-d67d-ecfa784b3ff9","executionInfo":{"status":"ok","timestamp":1584551162744,"user_tz":420,"elapsed":468335,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["# Fit on train data\n","y_tokenizer = y_tokenizer.fit(y_train)\n","classes = list(y_tokenizer.classes_)\n","print (f\"classes: {classes}\")"],"execution_count":52,"outputs":[{"output_type":"stream","text":["classes: ['Business', 'Sci/Tech', 'Sports', 'World']\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"yrLHd1i_8XAJ","colab_type":"code","outputId":"b2b25efb-0c4b-4b8f-bf21-8a5827a9056f","executionInfo":{"status":"ok","timestamp":1584551162744,"user_tz":420,"elapsed":468315,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["# Convert labels to tokens\n","y_train = y_tokenizer.transform(y_train)\n","y_val = y_tokenizer.transform(y_val)\n","y_test = y_tokenizer.transform(y_test)\n","print (f\"y_train[0]: {y_train[0]}\")"],"execution_count":53,"outputs":[{"output_type":"stream","text":["y_train[0]: 1\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"DY91F44BR15z","colab_type":"code","outputId":"1d60b27e-0ee0-4c6d-e205-47aee11488c4","executionInfo":{"status":"ok","timestamp":1584551162745,"user_tz":420,"elapsed":468297,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":51}},"source":["# Class weights\n","counts = np.bincount(y_train)\n","class_weights = {i: 1.0/count for i, count in enumerate(counts)}\n","print (f\"class counts: {counts},\\nclass weights: {class_weights}\")"],"execution_count":54,"outputs":[{"output_type":"stream","text":["class counts: [21675 21675 21675 21675],\n","class weights: {0: 4.61361014994233e-05, 1: 4.61361014994233e-05, 2: 4.61361014994233e-05, 3: 4.61361014994233e-05}\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"eoWQk0hO9bK2","colab_type":"text"},"source":["## Datasets"]},{"cell_type":"code","metadata":{"id":"GVxnbzgW8X1V","colab_type":"code","colab":{}},"source":["import math\n","from tensorflow.keras.preprocessing.sequence import pad_sequences\n","import torch\n","import torch.nn as nn\n","from torch.utils.data import Dataset\n","from torch.utils.data import DataLoader"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"i-ta3se1Cq-4","colab_type":"code","colab":{}},"source":["BATCH_SIZE = 64\n","FILTER_SIZES = [2, 3, 4]"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"dGMp4t7Wkdy0","colab_type":"code","outputId":"235c0521-465a-40b4-be17-cf384b8b8e95","executionInfo":{"status":"ok","timestamp":1584551166506,"user_tz":420,"elapsed":472036,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["# Set seed for reproducibility\n","torch.manual_seed(SEED)"],"execution_count":57,"outputs":[{"output_type":"execute_result","data":{"text/plain":["<torch._C.Generator at 0x7fd1e9274410>"]},"metadata":{"tags":[]},"execution_count":57}]},{"cell_type":"code","metadata":{"id":"YiJHwJPXkfBw","colab_type":"code","outputId":"7df2d03f-1509-4ffc-e858-4631663d8106","executionInfo":{"status":"ok","timestamp":1584551166509,"user_tz":420,"elapsed":472010,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n","print (device)"],"execution_count":58,"outputs":[{"output_type":"stream","text":["cuda\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"1w6wVKJe9fxk","colab_type":"code","colab":{}},"source":["class TextDataset(Dataset):\n","    \"\"\"Text dataset.\"\"\"\n","    def __init__(self, X, y, batch_size, max_filter_size):\n","        self.X = X\n","        self.y = y\n","        self.batch_size = batch_size\n","        self.max_filter_size = max_filter_size\n","    \n","    def __str__(self):\n","        return f\"<Dataset(N={len(self)}, batch_size={self.batch_size}, num_batches={self.get_num_batches()})>\"\n","\n","    def __len__(self):\n","        return len(self.y)\n","\n","    def __getitem__(self, index):\n","        X = self.X[index]\n","        y = self.y[index]\n","        return X, y\n","\n","    def get_num_batches(self):\n","        return math.ceil(len(self)/self.batch_size)\n","\n","    def collate_fn(self, batch):\n","        \"\"\"Processing on a batch.\"\"\"\n","        # Get inputs\n","        X = np.array(batch)[:, 0]\n","        y = np.array(batch)[:, 1]\n","\n","        # Pad inputs\n","        max_seq_len = max(self.max_filter_size, max([len(x) for x in X]))\n","        X = pad_sequences(X, padding=\"post\", maxlen=max_seq_len)\n","\n","        # Cast\n","        X = torch.LongTensor(X.astype(np.int32))\n","        y = torch.LongTensor(y.astype(np.int32))\n","\n","        return X, y\n","\n","    def generate_batches(self, shuffle=False, drop_last=False):\n","        dataloader = DataLoader(dataset=self, batch_size=self.batch_size, \n","                                collate_fn=self.collate_fn, \n","                                shuffle=shuffle, drop_last=drop_last)\n","        for (X, y) in dataloader:\n","            yield X, y"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"5T8mVj9d9hNI","colab_type":"code","outputId":"2ab38c2e-ecd4-473d-febd-86c694fdd582","executionInfo":{"status":"ok","timestamp":1584551166510,"user_tz":420,"elapsed":471987,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":51}},"source":["# Create datasets\n","train_set = TextDataset(X=X_train, y=y_train, batch_size=BATCH_SIZE, max_filter_size=max(FILTER_SIZES))\n","val_set = TextDataset(X=X_val, y=y_val, batch_size=BATCH_SIZE, max_filter_size=max(FILTER_SIZES))\n","test_set = TextDataset(X=X_test, y=y_test, batch_size=BATCH_SIZE, max_filter_size=max(FILTER_SIZES))\n","print (train_set)\n","print (train_set[0])"],"execution_count":60,"outputs":[{"output_type":"stream","text":["<Dataset(N=86700, batch_size=64, num_batches=1355)>\n","([2072, 1544, 92, 197, 563, 2, 2194, 69, 4804, 16], 1)\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"drbY5WDX9kcL","colab_type":"code","outputId":"1b436756-da4c-4be4-a5b2-0e77d4ba5010","executionInfo":{"status":"ok","timestamp":1584551166510,"user_tz":420,"elapsed":471969,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":51}},"source":["# Generate batch\n","batch_X, batch_y = next(iter(test_set.generate_batches()))\n","print (batch_X.shape)\n","print (batch_y.shape)"],"execution_count":61,"outputs":[{"output_type":"stream","text":["torch.Size([64, 15])\n","torch.Size([64])\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"581nl9EYFAsS","colab_type":"text"},"source":["# Embedding"]},{"cell_type":"markdown","metadata":{"id":"JbOzzfLNFCtW","colab_type":"text"},"source":["We can embed our inputs using the [embedding layer](https://pytorch.org/docs/stable/nn.html#torch.nn.Embedding)."]},{"cell_type":"code","metadata":{"id":"sqgPAuRLFC63","colab_type":"code","colab":{}},"source":["import torch\n","import torch.nn as nn"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"1tHb3v_KH53e","colab_type":"code","outputId":"7588b4cc-650b-4c2a-bfeb-75326a59400c","executionInfo":{"status":"ok","timestamp":1584551166511,"user_tz":420,"elapsed":471937,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":51}},"source":["# Input\n","x = torch.randint(high=10, size=(1,5)) # high = vocab size\n","print (x)\n","print (x.shape)"],"execution_count":63,"outputs":[{"output_type":"stream","text":["tensor([[6, 5, 6, 4, 2]])\n","torch.Size([1, 5])\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"FXUpmH7AFOJh","colab_type":"code","outputId":"dd4ec4ec-1555-4118-83df-9ac6a4a1d687","executionInfo":{"status":"ok","timestamp":1584551166511,"user_tz":420,"elapsed":471913,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["# Embedding layer\n","embeddings = nn.Embedding(embedding_dim=100,\n","                          num_embeddings=10, # vocab size\n","                          padding_idx=0) # which id is for padding\n","print (embeddings.weight.shape)"],"execution_count":64,"outputs":[{"output_type":"stream","text":["torch.Size([10, 100])\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"bVGWIgEGGmHn","colab_type":"code","outputId":"d070b211-a920-459d-8e06-fff0117252da","executionInfo":{"status":"ok","timestamp":1584551166512,"user_tz":420,"elapsed":471896,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["# Embed the input\n","embeddings(x).shape"],"execution_count":65,"outputs":[{"output_type":"execute_result","data":{"text/plain":["torch.Size([1, 5, 100])"]},"metadata":{"tags":[]},"execution_count":65}]},{"cell_type":"markdown","metadata":{"id":"WbO8HYjaGxZY","colab_type":"text"},"source":["Each id (token) in the input has been embedded using the embeddings. In the model below, we'll see how to preset our embeddings with our GloVe embeddings and how to choose whether to freeze (don't change during training) those embeddings or not. "]},{"cell_type":"markdown","metadata":{"id":"yV0AhZWOjXG0","colab_type":"text"},"source":["# Modeling"]},{"cell_type":"markdown","metadata":{"id":"pfhjWZRD94hK","colab_type":"text"},"source":["## Model"]},{"cell_type":"markdown","metadata":{"id":"eI5xEaMN-vT9","colab_type":"text"},"source":["Let's visualize the model's forward pass.\n","\n","1. We'll first tokenize our inputs (`batch_size`, `max_seq_len`).\n","2. Then we'll embed our tokenized inputs (`batch_size`, `max_seq_len`, `embedding_dim`).\n","3. We'll apply convolution via filters (`filter_size`, `vocab_size`, `num_filters`) followed by batch normalization. Our filters act as character level n-gram detecors. We have three different filter sizes (2, 3 and 4) and they will act as bi-gram, tri-gram and 4-gram feature extractors, respectivelyy. \n","4. We'll apply 1D global max pooling which will extract the most relevant information from the feature maps for making the decision.\n","5. We feed the pool outputs to a fully-connected (FC) layer (with dropout).\n","6. We use one more FC layer with softmax to derive class probabilities. "]},{"cell_type":"markdown","metadata":{"id":"zVmJGm8m-KIz","colab_type":"text"},"source":["<div align=\"left\">\n","<img src=\"https://raw.githubusercontent.com/madewithml/images/master/02_Basics/08_Embeddings/forward_pass.png\" width=\"1000\">\n","</div>"]},{"cell_type":"markdown","metadata":{"id":"JrVDcLC9kNMq","colab_type":"text"},"source":["The `FILTER_SIZES` are [2, 3, 4] which effectively act as bi-gram, tri-gram and 4th-gram feature extractors when applied to our text."]},{"cell_type":"code","metadata":{"id":"_I3dmAFtsfy6","colab_type":"code","colab":{}},"source":["import torch.nn.functional as F"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"UPP5ROd69mXC","colab_type":"code","colab":{}},"source":["class CNN(nn.Module):\n","    def __init__(self, embedding_dim, vocab_size, num_filters, \n","                 filter_sizes, hidden_dim, dropout_p, num_classes, \n","                 pretrained_embeddings=None, freeze_embeddings=False,\n","                 padding_idx=0):\n","        super(CNN, self).__init__()\n","\n","        # Filter sizes\n","        self.filter_sizes = filter_sizes\n","        \n","        # Initialize embeddings\n","        if pretrained_embeddings is None:\n","            self.embeddings = nn.Embedding(embedding_dim=embedding_dim,\n","                                          num_embeddings=vocab_size,\n","                                          padding_idx=padding_idx)\n","        else:\n","            pretrained_embeddings = torch.from_numpy(pretrained_embeddings).float()\n","            self.embeddings = nn.Embedding(embedding_dim=embedding_dim,\n","                                           num_embeddings=vocab_size,\n","                                           padding_idx=padding_idx,\n","                                           _weight=pretrained_embeddings)\n","        \n","        # Freeze embeddings or not\n","        if freeze_embeddings:\n","            self.embeddings.weight.requires_grad = False\n","        \n","        # Conv weights\n","        self.conv = nn.ModuleList(\n","            [nn.Conv1d(in_channels=embedding_dim, \n","                       out_channels=num_filters, \n","                       kernel_size=f) for f in filter_sizes])\n","     \n","        # FC weights\n","        self.dropout = nn.Dropout(dropout_p)\n","        self.fc1 = nn.Linear(num_filters*len(filter_sizes), hidden_dim)\n","        self.fc2 = nn.Linear(hidden_dim, num_classes)\n","\n","    def forward(self, x_in, channel_first=False, apply_softmax=False):\n","        \n","        # Embed\n","        x_in = self.embeddings(x_in)\n","\n","        # Rearrange input so num_channels is in dim 1 (N, C, L)\n","        if not channel_first:\n","            x_in = x_in.transpose(1, 2)\n","            \n","        # Conv outputs\n","        z = []\n","        max_seq_len = x_in.shape[2]\n","        for i, f in enumerate(self.filter_sizes):\n","            # `SAME` padding\n","            padding_left = int((self.conv[i].stride[0]*(max_seq_len-1) - max_seq_len + self.filter_sizes[i])/2)\n","            padding_right = int(math.ceil((self.conv[i].stride[0]*(max_seq_len-1) - max_seq_len + self.filter_sizes[i])/2))\n","\n","            # Conv + pool\n","            _z = self.conv[i](F.pad(x_in, (padding_left, padding_right)))\n","            _z = F.max_pool1d(_z, _z.size(2)).squeeze(2)\n","            z.append(_z)\n","        \n","        # Concat conv outputs\n","        z = torch.cat(z, 1)\n","\n","        # FC layers\n","        z = self.fc1(z)\n","        z = self.dropout(z)\n","        y_pred = self.fc2(z)\n","        \n","        if apply_softmax:\n","            y_pred = F.softmax(y_pred, dim=1)\n","        return y_pred"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"QBmYu6wjkgf0","colab_type":"text"},"source":["## GloVe embeddings"]},{"cell_type":"code","metadata":{"id":"x9uev5AGsuqq","colab_type":"code","colab":{}},"source":["def load_glove_embeddings(embeddings_file):\n","    \"\"\"Load embeddings from a file.\"\"\"\n","    embeddings = {}\n","    with open(embeddings_file, \"r\") as fp:\n","        for index, line in enumerate(fp):\n","            values = line.split()\n","            word = values[0]\n","            embedding = np.asarray(values[1:], dtype='float32')\n","            embeddings[word] = embedding\n","    return embeddings"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"tQHD-ThwWnjD","colab_type":"code","colab":{}},"source":["def make_embeddings_matrix(embeddings, word_index, embedding_dim):\n","    \"\"\"Create embeddings matrix to use in Embedding layer.\"\"\"\n","    embedding_matrix = np.zeros((len(word_index) + 1, embedding_dim))\n","    for word, i in word_index.items():\n","        embedding_vector = embeddings.get(word)\n","        if embedding_vector is not None:\n","            embedding_matrix[i] = embedding_vector\n","    return embedding_matrix"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"9WxP2GR3LmrO","colab_type":"code","outputId":"4758001e-8a27-4c7e-db4e-3323c765f9c8","executionInfo":{"status":"ok","timestamp":1584551176851,"user_tz":420,"elapsed":482193,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["# Create embeddings\n","embeddings_file = 'glove.6B.{0}d.txt'.format(EMBEDDING_DIM)\n","glove_embeddings = load_glove_embeddings(embeddings_file=embeddings_file)\n","embedding_matrix = make_embeddings_matrix(embeddings=glove_embeddings, \n","                                          word_index=X_tokenizer.word_index, \n","                                          embedding_dim=EMBEDDING_DIM)\n","print (f\"<Embeddings(words={embedding_matrix.shape[0]}, dim={embedding_matrix.shape[1]})>\")"],"execution_count":70,"outputs":[{"output_type":"stream","text":["<Embeddings(words=29795, dim=100)>\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"Sf-yZn5CzIex","colab_type":"text"},"source":["## Training"]},{"cell_type":"code","metadata":{"colab_type":"code","id":"FqYSeju9u0O2","colab":{}},"source":["class Trainer(object):\n","\n","    def __init__(self, **kwargs):\n","        self.__dict__ = kwargs\n","\n","    def train_loop(self, num_epochs):\n","        \"\"\"Training and validation steps.\"\"\"\n","        # Metrics\n","        self.train_loss = []\n","        self.train_acc = []\n","        self.val_loss = []\n","        self.val_acc = []\n","        best_val_loss = np.inf\n","\n","        # Epochs\n","        for epoch in range(num_epochs):\n","            # Steps\n","            self.train_step(epoch)\n","            self.val_step(epoch)\n","            print (f\"Epoch: {epoch} | train_loss: {self.train_loss[-1]:.2f}, train_acc: {self.train_acc[-1]:.1f}, val_loss: {self.val_loss[-1]:.2f}, val_acc: {self.val_acc[-1]:.1f}\")\n","\n","            # Early stopping\n","            if self.val_loss[-1] < best_val_loss:\n","                best_val_loss = self.val_loss[-1]\n","                patience = self.patience # reset patience\n","            else:\n","                patience -= 1\n","            if not patience: # 0\n","                print (\"Stopping early!\")\n","                break\n","\n","        return self.train_loss, self.train_acc, self.val_loss, self.val_acc, best_val_loss\n","\n","    def train_step(self, epoch):\n","        \"\"\"Training one epoch.\"\"\"\n","        # Set model to train mode\n","        self.model.train()\n","\n","        # Reset batch metrics\n","        running_train_loss = 0.0\n","        running_train_acc = 0.0\n","\n","        # Iterate over train batches\n","        for i, (X, y) in enumerate(self.train_set.generate_batches()):\n","\n","            # Set device\n","            X = X.to(self.device)\n","            y = y.to(self.device)\n","\n","            # Forward pass\n","            y_pred = self.model(X)\n","            loss = self.loss_fn(y_pred, y)\n","\n","            # Backward pass + optimize\n","            self.optimizer.zero_grad()\n","            loss.backward()\n","            self.optimizer.step()\n","\n","            # Metrics\n","            predictions = y_pred.max(dim=1)[1] # class\n","            accuracy = self.accuracy_fn(y_pred=predictions, y_true=y)\n","\n","            # Update batch metrics\n","            running_train_loss += (loss - running_train_loss) / (i + 1)\n","            running_train_acc += (accuracy - running_train_acc) / (i + 1)\n","        \n","        # Update epoch metrics\n","        self.train_loss.append(running_train_loss)\n","        self.train_acc.append(running_train_acc)\n","\n","        # Write to TensorBoard\n","        self.writer.add_scalar(tag='training loss', scalar_value=running_train_loss, global_step=epoch)\n","        self.writer.add_scalar(tag='training accuracy', scalar_value=running_train_acc, global_step=epoch)\n","\n","    def val_step(self, epoch):\n","        \"\"\"Validate one epoch.\"\"\"\n","        # Set model to eval mode\n","        self.model.eval()\n","\n","        # Reset batch metrics\n","        running_val_loss = 0.0\n","        running_val_acc = 0.0\n","\n","        # Iterate over val batches\n","        for i, (X, y) in enumerate(self.val_set.generate_batches()):\n","\n","            # Set device\n","            X = X.to(self.device)\n","            y = y.to(self.device)\n","\n","            # Forward pass\n","            with torch.no_grad():\n","                y_pred = self.model(X)\n","                loss = self.loss_fn(y_pred, y)\n","\n","            # Metrics\n","            predictions = y_pred.max(dim=1)[1] # class\n","            accuracy = self.accuracy_fn(y_pred=predictions, y_true=y)\n","\n","            # Update batch metrics\n","            running_val_loss += (loss - running_val_loss) / (i + 1)\n","            running_val_acc += (accuracy - running_val_acc) / (i + 1)\n","\n","        # Update epoch metrics\n","        self.val_loss.append(running_val_loss)\n","        self.val_acc.append(running_val_acc)\n","\n","        # Write to TensorBoard\n","        self.writer.add_scalar(tag='validation loss', scalar_value=running_val_loss, global_step=epoch)\n","        self.writer.add_scalar(tag='validation accuracy', scalar_value=running_val_acc, global_step=epoch)\n","\n","        # Adjust learning rate\n","        self.scheduler.step(running_val_loss)\n","\n","    def test_loop(self):\n","        \"\"\"Evalution of the test set.\"\"\"\n","        # Metrics\n","        running_test_loss = 0.0\n","        running_test_acc = 0.0\n","        y_preds = []\n","        y_targets = []\n","\n","        # Iterate over val batches\n","        for i, (X, y) in enumerate(self.test_set.generate_batches()):\n","\n","            # Set device\n","            X = X.to(self.device)\n","            y = y.to(self.device)\n","\n","            # Forward pass\n","            with torch.no_grad():\n","                y_pred = self.model(X)\n","                loss = self.loss_fn(y_pred, y)\n","\n","            # Metrics\n","            predictions = y_pred.max(dim=1)[1] # class\n","            accuracy = self.accuracy_fn(y_pred=predictions, y_true=y)\n","\n","            # Update batch metrics\n","            running_test_loss += (loss - running_test_loss) / (i + 1)\n","            running_test_acc += (accuracy - running_test_acc) / (i + 1)\n","\n","            # Store values\n","            y_preds.extend(predictions.cpu().numpy())\n","            y_targets.extend(y.cpu().numpy())\n","\n","        return running_test_loss, running_test_acc, y_preds, y_targets"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"C26maF-9Goit","colab_type":"text"},"source":["## Experiments"]},{"cell_type":"markdown","metadata":{"id":"eTWQcUJ_GrIx","colab_type":"text"},"source":["Once you have chosen your embeddings, you can choose to freeze them or continue to train them using the supervised data (this could lead to overfitting). In this example, we will do three experiments: \n","* frozen GloVe embeddings\n","* fine-tuned (unfrozen) GloVe embeddings\n","* randomly initialized embeddings"]},{"cell_type":"code","metadata":{"id":"geKOPVzVK6S9","colab_type":"code","colab":{}},"source":["import matplotlib.pyplot as plt\n","from torch.optim import Adam\n","from torch.optim.lr_scheduler import ReduceLROnPlateau\n","from torch.utils.tensorboard import SummaryWriter\n","from torchsummary import summary\n","%load_ext tensorboard"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"64iPmq2lDv2h","colab_type":"code","colab":{}},"source":["EMBEDDING_DIM = 100\n","NUM_FILTERS = 50\n","HIDDEN_DIM = 100\n","DROPOUT_P = 0.1\n","LEARNING_RATE = 1e-3\n","PATIENCE = 3\n","NUM_EPOCHS = 10"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"To_CB7ibLesP","colab_type":"text"},"source":["### GloVe embeddings (frozen)"]},{"cell_type":"code","metadata":{"id":"oT9w__AMkqfG","colab_type":"code","colab":{}},"source":["FREEZE_EMBEDDINGS = True"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"yg13AyoUkqcJ","colab_type":"code","outputId":"9c858ee3-844d-40f1-ac3f-8dd09793586a","executionInfo":{"status":"ok","timestamp":1584551185672,"user_tz":420,"elapsed":490964,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":204}},"source":["# Initialize model\n","glove_frozen_model = CNN(embedding_dim=EMBEDDING_DIM,\n","                         vocab_size=vocab_size,\n","                         num_filters=NUM_FILTERS,\n","                         filter_sizes=FILTER_SIZES,\n","                         hidden_dim=HIDDEN_DIM,\n","                         dropout_p=DROPOUT_P,\n","                         num_classes=len(classes),\n","                         pretrained_embeddings=embedding_matrix,\n","                         freeze_embeddings=FREEZE_EMBEDDINGS).to(device)\n","model = glove_frozen_model\n","print (model.named_parameters)\n","# summary(model, input_size=(10, vocab_size)) # bug: can't make inputs into LongTensor"],"execution_count":75,"outputs":[{"output_type":"stream","text":["<bound method Module.named_parameters of CNN(\n","  (embeddings): Embedding(29795, 100, padding_idx=0)\n","  (conv): ModuleList(\n","    (0): Conv1d(100, 50, kernel_size=(2,), stride=(1,))\n","    (1): Conv1d(100, 50, kernel_size=(3,), stride=(1,))\n","    (2): Conv1d(100, 50, kernel_size=(4,), stride=(1,))\n","  )\n","  (dropout): Dropout(p=0.1, inplace=False)\n","  (fc1): Linear(in_features=150, out_features=100, bias=True)\n","  (fc2): Linear(in_features=100, out_features=4, bias=True)\n",")>\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"n-OQ-PRfJFdR","colab_type":"code","colab":{}},"source":["# Loss\n","weights = torch.Tensor([class_weights[key] for key in sorted(class_weights.keys())]).to(device)\n","loss_fn = nn.CrossEntropyLoss(weight=weights)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"gi9DSAYGkuBW","colab_type":"code","colab":{}},"source":["# Accuracy\n","def accuracy_fn(y_pred, y_true):\n","    n_correct = torch.eq(y_pred, y_true).sum().item()\n","    accuracy = (n_correct / len(y_pred)) * 100\n","    return accuracy"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"olAw2yp8t4bu","colab_type":"code","colab":{}},"source":["# Optimizer\n","optimizer = Adam(model.parameters(), lr=LEARNING_RATE) \n","scheduler = ReduceLROnPlateau(optimizer, mode='min', factor=0.1, patience=3)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"-uvFsxH34T7K","colab_type":"code","colab":{}},"source":["# Create writer to store values \n","log_dir = 'tensorboard/glove_frozen'\n","!rm -rf log_dir # remove if it already exists\n","writer = SummaryWriter(log_dir=log_dir)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"colab_type":"code","id":"aoM7tPq_u0AL","outputId":"2f124a83-b9c0-4263-88d2-e5b98985ce53","executionInfo":{"status":"ok","timestamp":1584551214873,"user_tz":420,"elapsed":520124,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":119}},"source":["# Train\n","trainer = Trainer(train_set=train_set, val_set=val_set, test_set=test_set, \n","                  model=model, optimizer=optimizer, scheduler=scheduler, \n","                  loss_fn=loss_fn, accuracy_fn=accuracy_fn, patience=PATIENCE, \n","                  writer=writer, device=device)\n","train_loss, train_acc, val_loss, val_acc, best_val_loss = trainer.train_loop(num_epochs=NUM_EPOCHS)"],"execution_count":80,"outputs":[{"output_type":"stream","text":["Epoch: 0 | train_loss: 0.44, train_acc: 84.5, val_loss: 0.40, val_acc: 86.0\n","Epoch: 1 | train_loss: 0.36, train_acc: 87.7, val_loss: 0.39, val_acc: 86.2\n","Epoch: 2 | train_loss: 0.31, train_acc: 89.0, val_loss: 0.41, val_acc: 85.6\n","Epoch: 3 | train_loss: 0.28, train_acc: 90.2, val_loss: 0.43, val_acc: 85.1\n","Epoch: 4 | train_loss: 0.25, train_acc: 91.3, val_loss: 0.46, val_acc: 84.9\n","Stopping early!\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"dUVkeDbNqO7V","colab_type":"text"},"source":["### Fine-tuned GloVe embeddings (unfrozen)"]},{"cell_type":"code","metadata":{"id":"eubLrHydkt_J","colab_type":"code","colab":{}},"source":["# Arguments\n","FREEZE_EMBEDDINGS = False"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"IGeZwoy9qUpa","colab_type":"code","outputId":"be5a4a64-5900-4854-d625-213be3b360e6","executionInfo":{"status":"ok","timestamp":1584551214875,"user_tz":420,"elapsed":520100,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":204}},"source":["# Initialize model\n","glove_finetuned_model = CNN(embedding_dim=EMBEDDING_DIM,\n","                         vocab_size=vocab_size,\n","                         num_filters=NUM_FILTERS,\n","                         filter_sizes=FILTER_SIZES,\n","                         hidden_dim=HIDDEN_DIM,\n","                         dropout_p=DROPOUT_P,\n","                         num_classes=len(classes),\n","                         pretrained_embeddings=embedding_matrix,\n","                         freeze_embeddings=FREEZE_EMBEDDINGS).to(device)\n","model = glove_finetuned_model\n","print (model.named_parameters)"],"execution_count":82,"outputs":[{"output_type":"stream","text":["<bound method Module.named_parameters of CNN(\n","  (embeddings): Embedding(29795, 100, padding_idx=0)\n","  (conv): ModuleList(\n","    (0): Conv1d(100, 50, kernel_size=(2,), stride=(1,))\n","    (1): Conv1d(100, 50, kernel_size=(3,), stride=(1,))\n","    (2): Conv1d(100, 50, kernel_size=(4,), stride=(1,))\n","  )\n","  (dropout): Dropout(p=0.1, inplace=False)\n","  (fc1): Linear(in_features=150, out_features=100, bias=True)\n","  (fc2): Linear(in_features=100, out_features=4, bias=True)\n",")>\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"oUaEr92PqUml","colab_type":"code","colab":{}},"source":["# Optimizer\n","optimizer = Adam(model.parameters(), lr=LEARNING_RATE) \n","scheduler = ReduceLROnPlateau(optimizer, mode='min', factor=0.1, patience=3)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"3qf5oYZd4RZ3","colab_type":"code","colab":{}},"source":["# Create writer to store values \n","log_dir = 'tensorboard/glove_finetuned'\n","!rm -rf log_dir # remove if it already exists\n","writer = SummaryWriter(log_dir=log_dir)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"NpyhLUK2qUjb","colab_type":"code","outputId":"e58bcfbe-ef4c-46f4-baa6-79710abacb0f","executionInfo":{"status":"ok","timestamp":1584551244557,"user_tz":420,"elapsed":549757,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":102}},"source":["# Train\n","trainer = Trainer(train_set=train_set, val_set=val_set, test_set=test_set, \n","                  model=model, optimizer=optimizer, scheduler=scheduler, \n","                  loss_fn=loss_fn, accuracy_fn=accuracy_fn, patience=PATIENCE, \n","                  writer=writer, device=device)\n","train_loss, train_acc, val_loss, val_acc, best_val_loss = trainer.train_loop(num_epochs=NUM_EPOCHS)"],"execution_count":85,"outputs":[{"output_type":"stream","text":["Epoch: 0 | train_loss: 0.41, train_acc: 85.6, val_loss: 0.36, val_acc: 87.7\n","Epoch: 1 | train_loss: 0.27, train_acc: 90.7, val_loss: 0.36, val_acc: 87.8\n","Epoch: 2 | train_loss: 0.19, train_acc: 93.7, val_loss: 0.42, val_acc: 87.2\n","Epoch: 3 | train_loss: 0.13, train_acc: 95.8, val_loss: 0.52, val_acc: 86.2\n","Stopping early!\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"Y8JzMrcv_p8a","colab_type":"text"},"source":["### Randomly initialized embeddings"]},{"cell_type":"code","metadata":{"id":"TnLSYV0WKo8x","colab_type":"code","colab":{}},"source":["# Arguments\n","FREEZE_EMBEDDINGS = False"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"wD4sRUS5_lwq","colab_type":"code","outputId":"5109d441-87fb-42eb-e96f-1e8dd49a3912","executionInfo":{"status":"ok","timestamp":1584551244558,"user_tz":420,"elapsed":549724,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":204}},"source":["randomly_initialized_model = CNN(embedding_dim=EMBEDDING_DIM,\n","                               vocab_size=vocab_size,\n","                               num_filters=NUM_FILTERS,\n","                               filter_sizes=FILTER_SIZES,\n","                               hidden_dim=HIDDEN_DIM,\n","                               dropout_p=DROPOUT_P,\n","                               num_classes=len(classes),\n","                               pretrained_embeddings=None,\n","                               freeze_embeddings=FREEZE_EMBEDDINGS).to(device)\n","model = randomly_initialized_model\n","print (model.named_parameters)"],"execution_count":87,"outputs":[{"output_type":"stream","text":["<bound method Module.named_parameters of CNN(\n","  (embeddings): Embedding(29795, 100, padding_idx=0)\n","  (conv): ModuleList(\n","    (0): Conv1d(100, 50, kernel_size=(2,), stride=(1,))\n","    (1): Conv1d(100, 50, kernel_size=(3,), stride=(1,))\n","    (2): Conv1d(100, 50, kernel_size=(4,), stride=(1,))\n","  )\n","  (dropout): Dropout(p=0.1, inplace=False)\n","  (fc1): Linear(in_features=150, out_features=100, bias=True)\n","  (fc2): Linear(in_features=100, out_features=4, bias=True)\n",")>\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"Ucn3tYq1_sE1","colab_type":"code","colab":{}},"source":["# Optimizer\n","optimizer = Adam(model.parameters(), lr=LEARNING_RATE) \n","scheduler = ReduceLROnPlateau(optimizer, mode='min', factor=0.1, patience=3)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"by1-D8cN4W2u","colab_type":"code","colab":{}},"source":["# Create writer to store values \n","log_dir = 'tensorboard/randomly_initialized'\n","!rm -rf log_dir # remove if it already exists\n","writer = SummaryWriter(log_dir=log_dir)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"F7bTmNdCJA0g","colab_type":"code","outputId":"d377761a-6207-4747-8570-680c4683a55f","executionInfo":{"status":"ok","timestamp":1584551282592,"user_tz":420,"elapsed":587728,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":119}},"source":["# Train\n","trainer = Trainer(train_set=train_set, val_set=val_set, test_set=test_set, \n","                  model=model, optimizer=optimizer, scheduler=scheduler, \n","                  loss_fn=loss_fn, accuracy_fn=accuracy_fn, patience=PATIENCE, \n","                  writer=writer, device=device)\n","train_loss, train_acc, val_loss, val_acc, best_val_loss = trainer.train_loop(num_epochs=NUM_EPOCHS)"],"execution_count":90,"outputs":[{"output_type":"stream","text":["Epoch: 0 | train_loss: 0.81, train_acc: 67.8, val_loss: 0.66, val_acc: 76.1\n","Epoch: 1 | train_loss: 0.46, train_acc: 83.6, val_loss: 0.62, val_acc: 79.2\n","Epoch: 2 | train_loss: 0.32, train_acc: 89.2, val_loss: 0.65, val_acc: 80.0\n","Epoch: 3 | train_loss: 0.22, train_acc: 92.6, val_loss: 0.66, val_acc: 81.2\n","Epoch: 4 | train_loss: 0.17, train_acc: 94.6, val_loss: 0.77, val_acc: 80.7\n","Stopping early!\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"vskwiiI3V3S6","colab_type":"text"},"source":["## Evaluation"]},{"cell_type":"markdown","metadata":{"id":"6tO2hX8OLQ5s","colab_type":"text"},"source":["Looks like fine-tuned glove embeddings had the best test performance (based on validation accuracy) so let's do proper evaluation and inference with that strategy."]},{"cell_type":"code","metadata":{"id":"Itq7lT9qV9Y8","colab_type":"code","colab":{}},"source":["import io\n","import itertools\n","import json\n","import matplotlib.pyplot as plt\n","from sklearn.metrics import classification_report\n","from sklearn.metrics import confusion_matrix\n","from sklearn.metrics import precision_recall_fscore_support"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"NNeyYs3tW3VN","colab_type":"code","colab":{}},"source":["def plot_confusion_matrix(y_true, y_pred, classes, cmap=plt.cm.Blues):\n","    \"\"\"Plot a confusion matrix using ground truth and predictions.\"\"\"\n","    # Confusion matrix\n","    cm = confusion_matrix(y_true, y_pred)\n","    cm_norm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n","\n","    #  Figure\n","    fig = plt.figure()\n","    ax = fig.add_subplot(111)\n","    cax = ax.matshow(cm, cmap=plt.cm.Blues)\n","    fig.colorbar(cax)\n","\n","    # Axis\n","    plt.title(\"Confusion matrix\")\n","    plt.ylabel(\"True label\")\n","    plt.xlabel(\"Predicted label\")\n","    ax.set_xticklabels([''] + classes)\n","    ax.set_yticklabels([''] + classes)\n","    ax.xaxis.set_label_position('bottom') \n","    ax.xaxis.tick_bottom()\n","\n","    # Values\n","    thresh = cm.max() / 2.\n","    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n","        plt.text(j, i, f\"{cm[i, j]:d} ({cm_norm[i, j]*100:.1f}%)\",\n","                 horizontalalignment=\"center\",\n","                 color=\"white\" if cm[i, j] > thresh else \"black\")\n","\n","    # Display\n","    plt.show()"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"smP8T1bEW3fH","colab_type":"code","colab":{}},"source":["def get_performance(y_true, y_pred, classes):\n","    \"\"\"Per-class performance metrics. \"\"\"\n","    performance = {'overall': {}, 'class': {}}\n","    metrics = precision_recall_fscore_support(y_true, y_pred)\n","\n","    # Overall performance\n","    performance['overall']['precision'] = np.mean(metrics[0])\n","    performance['overall']['recall'] = np.mean(metrics[1])\n","    performance['overall']['f1'] = np.mean(metrics[2])\n","    performance['overall']['num_samples'] = np.float64(np.sum(metrics[3]))\n","\n","    # Per-class performance\n","    for i in range(len(classes)):\n","        performance['class'][classes[i]] = {\n","            \"precision\": metrics[0][i],\n","            \"recall\": metrics[1][i],\n","            \"f1\": metrics[2][i],\n","            \"num_samples\": np.float64(metrics[3][i])\n","        }\n","\n","    return performance"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"avmwpr5syKHY","colab_type":"code","outputId":"b4486135-c371-49f8-e693-7199e677e630","executionInfo":{"status":"ok","timestamp":1584551600028,"user_tz":420,"elapsed":968,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["# Test loop\n","model = glove_finetuned_model\n","test_loss, test_acc, y_preds, y_targets = trainer.test_loop()\n","print (f\"test_loss: {test_loss:.2f}, test_acc: {test_acc:.1f}\")"],"execution_count":125,"outputs":[{"output_type":"stream","text":["test_loss: 0.76, test_acc: 80.5\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"qdAj6KyCU88E","colab_type":"code","outputId":"120b0a2b-696e-4c39-e79d-78f55c16cd02","executionInfo":{"status":"ok","timestamp":1584551282594,"user_tz":420,"elapsed":587669,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":595}},"source":["# Class performance\n","performance = get_performance(y_true=y_targets, y_pred=y_preds, classes=classes)\n","print (json.dumps(performance, indent=4))"],"execution_count":95,"outputs":[{"output_type":"stream","text":["{\n","    \"overall\": {\n","        \"precision\": 0.8076232111724061,\n","        \"recall\": 0.8049444444444445,\n","        \"f1\": 0.8027832881817765,\n","        \"num_samples\": 18000.0\n","    },\n","    \"class\": {\n","        \"Business\": {\n","            \"precision\": 0.8384386012469504,\n","            \"recall\": 0.6873333333333334,\n","            \"f1\": 0.7554035901819515,\n","            \"num_samples\": 4500.0\n","        },\n","        \"Sci/Tech\": {\n","            \"precision\": 0.7947049584922594,\n","            \"recall\": 0.7871111111111111,\n","            \"f1\": 0.7908898068549738,\n","            \"num_samples\": 4500.0\n","        },\n","        \"Sports\": {\n","            \"precision\": 0.7800373134328358,\n","            \"recall\": 0.9291111111111111,\n","            \"f1\": 0.8480730223123731,\n","            \"num_samples\": 4500.0\n","        },\n","        \"World\": {\n","            \"precision\": 0.817311971517579,\n","            \"recall\": 0.8162222222222222,\n","            \"f1\": 0.8167667333778075,\n","            \"num_samples\": 4500.0\n","        }\n","    }\n","}\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"nRbPfqgZWaof","colab_type":"code","outputId":"7da8a938-bb5e-4ac5-f8b3-55d147a7fec4","executionInfo":{"status":"ok","timestamp":1584551282595,"user_tz":420,"elapsed":587650,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":598}},"source":["# Confusion matrix\n","plt.rcParams[\"figure.figsize\"] = (7,7)\n","plot_confusion_matrix(y_targets, y_preds, classes=classes)\n","print (classification_report(y_targets, y_preds))"],"execution_count":96,"outputs":[{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAc8AAAGKCAYAAABq7cr0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjAsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8GearUAAAgAElEQVR4nOzdd3wURRvA8d+TAgm9FwFButRA6L03\naQqCihRFAUUREGmiKBYQFVCkiAVFfAEFlC5Neu8gCNKlhRZ6epj3j90cF3JpeAmEPF8/9zE7OzM7\nt1ny3MzOzokxBqWUUkolnMf9boBSSimV0mjwVEoppRJJg6dSSimVSBo8lVJKqUTS4KmUUkolktf9\nboBSSqmHh2emgsZEBLulLhN8cakxpplbKnMzDZ5KKaXcxkQEk7ZEB7fUFbJ7Qg63VJQENHgqpZRy\nIwF5+O8IPvzvUCmllHIzDZ5KxUNEfEVkgYhcE5Ff/0M9nURkmTvbdr+ISG0ROXS/26EeQAKIuOf1\nANPgqR4aIvKciGwXkZsick5ElohILTdU3R7IDWQ3xjx9r5UYY342xjRxQ3uSlIgYESkaVx5jzDpj\nTInkapNKYcTDPa8H2IPdOqUSSET6A+OAj7EC3aPARKCNG6ovCPxjjIlwQ10pnojoXAmV6mnwVCme\niGQGRgC9jTFzjTG3jDHhxpgFxpi37DxpRWSciJy1X+NEJK29r56InBaRN0Xkgt1rfcHe9z7wLtDR\n7tF2F5H3RGS60/EL2b01L3u7m4gcE5EbInJcRDo5pa93KldDRLbZw8HbRKSG077VIvKBiGyw61km\nIi5nHjq1f6BT+9uKSAsR+UdEAkVkqFP+KiKySUSu2nm/EpE09r61drY99vvt6FT/IBEJAKZGpdll\nitjHqGhvPyIiF0Wk3n/6xaqUS4dtlUoRqgM+wG9x5HkbqAb4AeWBKsAwp/15gMxAPqA7MEFEshpj\nhmP1ZmcZYzIYY76LqyEikh74EmhujMkI1AB2u8iXDVhk580OjAEWiUh2p2zPAS8AuYA0wIA4Dp0H\n6xzkwwr23wDPA/5AbeAdEXnMzhsJ9ANyYJ27hsCrAMaYOnae8vb7neVUfzasXngP5wMbY44Cg4Dp\nIpIOmAr8aIxZHUd71UNLdNhWqRQiO3ApnmHVTsAIY8wFY8xF4H2gs9P+cHt/uDFmMXATuNd7ereB\nMiLia4w5Z4zZ7yLPE8BhY8xPxpgIY8wM4CDQyinPVGPMP8aYYOAXrMAfm3DgI2NMODATKzB+YYy5\nYR//ANaHBowxO4wxm+3jngC+Buom4D0NN8aE2u2JxhjzDXAE2ALkxfqwotRDS4OnehhcBnLEcy/u\nEeCk0/ZJO81Rx13BNwjIkNiGGGNuAR2BXsA5EVkkIiUT0J6oNuVz2g5IRHsuG2Mi7Z+jgtt5p/3B\nUeVFpLiILBSRABG5jtWzju9h9IvGmJB48nwDlAHGG2NC48mrHmY6bKtUirAJCAXaxpHnLNaQY5RH\n7bR7cQtI57Sdx3mnMWapMaYxVg/sIFZQia89UW06c49tSoxJWO0qZozJBAzFesAgLiaunSKSAWvC\n1nfAe/awtEqNBB22VSolMMZcw7rPN8GeKJNORLxFpLmIjLazzQCGiUhOe+LNu8D02OqMx26gjog8\nak9WGhK1Q0Ryi0gb+95nKNbw720XdSwGituP13iJSEegFLDwHtuUGBmB68BNu1f8yl37zwOFE1nn\nF8B2Y8xLWPdyJ//nVir1ANPgqR4KxpjPgf5Yk4AuAqeA14Df7SwfAtuBvcA+YKeddi/HWg7Msuva\nQfSA52G34ywQiHUv8e7ghDHmMtASeBNr2Hkg0NIYc+le2pRIA7AmI93A6hXPumv/e8CP9mzceBcp\nFZE2QDPuvM/+QMWoWcYqtXHTkO0DPmwrxsQ5GqOUUkolmEeGvCZtuRfcUlfIppE7jDGV4sojIp5Y\nH4zPGGNa2rPKZ2JNJNwBdDbGhNmPpk3DmoF+GehoT5hDRIZgzbKPBPoYY5bG1zbteSqllErJ3gD+\ndtr+BBhrjCkKXMEKitj/v2Knj7XzISKlgGeA0lgjKBPtgBwnDZ5KKaXcK5mGbUUkP9ZjX9/a2wI0\nAGbbWX7kzkTCNvY29v6Gdv42wEz7MazjWI9cVYnv2LrMllJKKTdy61eS5RCR7U7bU4wxU5y2x2HN\nF8hob2cHrjo9dnaaO49/5cOaC4ExJkJErtn58wGbnep0LhMrDZ5KKaUeVJdiu+cpIi2BC8aYHfdj\nKUgNnkoppdwn6ivJkl5NoLWItMBamjIT1iNTWUTEy+595ufOs9NngALAaXtBlcxYE4ei0qM4l4mV\n3vNUSinlXsmwSIIxZogxJr8xphDWhJ8/jTGdgFVYXyMI0BWYZ/88397G3v+nsR43mQ88Y395xGNA\nMWBrfG9Re55KKaUeJoOAmSLyIbALa9Ur7P//JCJHsJ7BfgbAGLNfRH7BWv85AuvbmSJjVhudPuep\nlFLKbTwy5jNpK/Z0S10ha4fH+5zn/aI9T6WUUu7l8WCvDuQOes9TKaWUSiTteSqllHKfqG9Vechp\n8FRKKeVeD/ii7u7w8H88UEoppdxMe55KKaXcyK3L8z2wNHgqpZRyLx22VUoppdTdtOeplFLKvXTY\nVimllEqEBH4XZ0qnwVMppZR7ac9TAXj6ZjbemXPd72akOMXyZIw/k4rBIxV8alcPjlP/niTw8iW9\n6BJJg2cCeGfORYHOX97vZqQ48wbWu99NSJF803je7yaoVKRZverurzQVfADU4KmUUsqNUsdzng//\nO1RKKaXcTHueSiml3EuHbZVSSqlESCXfqvLwv0OllFLKzbTnqZRSyo1Sx4QhDZ5KKaXcKxXc83z4\nPx4opZRSbqY9T6WUUu6lw7ZKKaVUIumwrVJKKaXupj1PpZRS7iM621YppZRKPB22VUoppdTdtOep\nlFLKrSQV9Dw1eCqllHIbIXUETx22VUoppRJJe55KKaXcR+zXQ06Dp1JKKTcSHbZVSimlVEza81RK\nKeVWqaHnqcFTKaWUW6WG4KnDtkoppVQiac9TKaWUW2nPUymllEoMceMrrsOI+IjIVhHZIyL7ReR9\nO/0HETkuIrvtl5+dLiLypYgcEZG9IlLRqa6uInLYfnVNyNvUnqdSSqmUKBRoYIy5KSLewHoRWWLv\ne8sYM/uu/M2BYvarKjAJqCoi2YDhQCXAADtEZL4x5kpcB9eep1JKKbcR+zlPd7ziYiw37U1v+2Xi\nKNIGmGaX2wxkEZG8QFNguTEm0A6Yy4Fm8b1PDZ5KKaXcyo3BM4eIbHd69bjrOJ4ishu4gBUAt9i7\nPrKHZseKSFo7LR9wyqn4aTsttvQ46bCtUkqpB9UlY0yl2HYaYyIBPxHJAvwmImWAIUAAkAaYAgwC\nRri7YdrzVEop5VbJMWzrzBhzFVgFNDPGnLOHZkOBqUAVO9sZoIBTsfx2WmzpcdLgqZRSyq2SI3iK\nSE67x4mI+AKNgYP2fUzEqqAt8JddZD7QxZ51Ww24Zow5BywFmohIVhHJCjSx0+KkwfMepfHyYPbr\n1ZnfryaL36xFnyZFHfvyZ/Vl9uvVWTGoDuM6+eHtaV0Ej2Tx4ccelVnQvybTe1UhT2YfR/rvb9Rw\n1PVstQIujwkwvnMFCmTzBcDbU/igXWmWDazDH2/VpmnZ3ADkzeLDTz2rMK9vTRb0r0ndkjlj1PNY\nzvTM71fT8dr1QWO61SoEwFstSrCgf01GP1POkb91xUcc+wGK58nAJx3L3tvJc1LHvyTN61amZf2q\ntGlc05F+YN8e2jWv60jfs3MbAPNmz6RF3So0r1uZ9i3q8/dfe13Wa4yh01PNuXHjOgCD3uhJ5VIF\naVYn+gjQyPeG0riGHy3qVqFX145cv3bVZX1Tp0ygWZ1KNKvtz9Svv3KkfzJiGC3qVuHN3i850n7/\ndUa0PIcO/MVbr0e7VeN2kZGRNK5dhS4d2zrSvp8ykRoVHueRLGm5fPmSI/2PRfNpWMOfRrUq06xe\ndbZs2uCyzuDgYJ5q0YjIyEgAPhw+lPrVK1C/egXmzf3VZZmvvxpH3arlaVjDnw6tm3L635MAHDl8\niKZ1q9Gwhj/bt24GICIigg5tmhEUFOQo3+vF5zl29PB/OxmJkJjzNveXGTSs4U+DGhVp1aQu+/fF\nfu093aopN65b1963k8ZTv3oF6lXz45uJX7osE1vdly9dpE2z+tSvXoElC+c58nd7th0B5846tt8f\nNoj1a1bd+4lImfICq0RkL7AN657nQuBnEdkH7ANyAB/a+RcDx4AjwDfAqwDGmEDgA7uObcAIOy1O\nGjzvUVjEbbp8vZXWYzfQeuwG6pTIid+jWQB464kSTF17gkafrOV6cDhPV7GC4eCWJfl9x1lajdnA\nV8uP8Gbz4gBcvBFKh68203rsBtqP30SP+oXJlSltjGMWzZ0BDw84FRgMwCsNixB4M4wmo9fS/LN1\nbD1q/b5fbViEJXsDaDNuA/2m7+G9J0vFqOv4xVuOtrcdt4Hg8EiW/RVABh8vSufLRKsxGwiPuE3x\nPBlI6+VBu0r5mL7xpKP8PwE3yZ3Zh7xZfP7zufx57hIWrtrCvOV3/oh/MmIYrw8YysJVW+g76B0+\nGTEMgPyPFmLGvKUsWbON194czNsDXnNZ5+oVf/B46bJkzJgJgHbPdGbqzN9j5KtVtwFL1m5n8Zqt\nPFakGJO++CxGnkN/72fW9Kn89sdaFq7awp/LlnDi2FFuXL/G/r27WbxmK2nSeHPowF+EBAcze+ZP\nPP9iT0f5EqXKEHD2DGdPn4pRt7t8O2k8xUqUjJZWuWoNZv2+hPwFCkZLr123ASs2bGfF+m2M+WoK\nA/r0clnnzOk/0KJVGzw9PVmxdDH79uxi+bptLFqxnsnjxzqCg7My5fxYsmoTKzfu4Ik2T/HB8KEA\nTJ/6LSNGfc5Pv85j8vixAPz43de06/Ac6dKlc5Tv+mIPJn4x5j+di8RIzHkrULAQcxav4M+NO+n3\n1hAG9n3VZZ0rly2hVJmyZMyUiYMH9vPztO9ZtHIDK9ZvZ/nSxRw/diRGmdjq/m32LLq88DKLVm7g\n20nWB7JlSxZSppwfefI+4ijfvcerfDXu0/90LtwmmZ7zNMbsNcZUMMaUM8aUMcaMsNMbGGPK2mnP\nR83ItYdyextjitj7tzvV9b0xpqj9mpqQt6nB8z8ICrM+kXt5Cl4egjHWLOlqRbPzx74AAObuOEOj\n0rkAK/htOnIZgM1HA2lU2uophkcawiJvA1aP1iOW4YrWFR5h5f4Lju32lfMz+c9jABgDV4LCHfsy\npLXmgmXw9eLC9dA430eNYjn493IQZ6+GYIzBy+4p+6bxJCLS8FK9x/hpw0kibkefBb7qwAVa+uWN\ns+57JSLcvHEDgBvXr5Mrj3Uc/yrVyJwlKwAV/KsQcNb1rYl5c2bRqFlLx3aV6rXIkiVbjHy16zfC\ny8s6V37+lV3Wd/TwIfwqVsI3XTq8vLyoUqMWSxfNQzw8iIgIxxhDcHAwXt7efDNxHF2698Lb2zta\nHQ2atmDBb657a//V2TOnWblsCc91fiFaetnyfhQoWChG/vQZMjiGxIKCbsU6PDb315k0bdEKgH8O\n/U21GrXx8vIiXfr0PF66LKtWLotRpmadeo5gWLFSVc7Z59PL25vg4CCCg4Pw8vbm2tWrLP9jEU8/\n+3y08lVr1GLd6pVEREQk7iTcg8Set8pVq5PFvvYqVr7z3u7mfN4O/3OQCv5VSGdfO9Vr1mHxgpgf\n4mKr29s+b2FhoXh6ehAREcG3k8bz6htvRiuf/9GCXAkM5ML5gMSdhCSS3Pc874ckC54iEinW6g57\nRGSniNS4x3p6iUgXd7fPHTwE5veryebhDdlw+DJ7Tl0jazpvbgSHE2kHmoCrIeS2h2cPnrvhGFpt\nUiY3GXy8yJLO+iObJ7MPC/rXZO3b9Zmy+pjLgOdfKCt/nb4GQEYf6w9+32bF+P2NGnz5vB/ZM6QB\n4MtlR2hd8RHWvV2fb1+sxIjfD8T5Pp4on5eFu6whoFuhkaw5eJH5/Wpy4XooN0IiKF8gCyucgnaU\nfaevUemxmAEpMUSEbh1a0bpRDWZM+86RPuzD0Yx6fyg1/Yox6r0hvPV2zMlyv/z8I3UbNnFZ746t\nmyhTvkKi2jJ7xjSX9RUvWYptmzdyJfAywUFBrFmxlHNnT5MhQ0bqNmxKqwbVyJUrDxkzZWLPzm00\nadE6Rh1ly1dk+xbXw6P/1fAhAxg2YiQeHgn/57xkwTxqVy5Llw5tGfPVlBj7w8LC+PfEcUcQKVWm\nHKtWLiMoKIjLly+xcd3qeHvSM6ZPpUGjpgB0e6kX4z8fTd9XXqJP/0GM+/Rj+vQfFKPNHh4eFCpc\nhAOxDMe7072ctygzfppKffu93W3b5k2U87MWryn5eCm2blpPYOBlgoKC+HP5H5w9fTrBdT/Z/hmW\nLl7AM21b8Pqbg/jh28m069gpWm89StnyFdi2eVOi34u6N0n5qEqwMSZqWaSmwEigbmIrMcZMdnfD\n3OW2gdZjN5DRx4uJXStSLHcGLt2IvZc3auFBhrctxVOV8rPtWCABV0PuBNlrIbQas4FcmdIysWtF\n/tgbwOWbYdHK58yUlsBbVpqXh5A3iy+7Tlxl5IKDvFCnEINbluStmXtpWSEvc7ef5vu1J/ArmIXP\nni1Pi8/XYVw8PuztKTQonYvPlhxypH2z+jjfrD4OwEftyzBu2WGerpKfWsVzcOjcDSauPArA5Zth\nLoeXE2PWghXkyZuPSxcv0PXpVhQpVoIq1Wvx8w/fMGzEaJq1asuieXMY3PcVfpqzyFFu0/o1/Pq/\nH5m1YIXLeq9duUKGDBkT3I4JYz/B09OLNu2fibGvaPGS9Hy9P107tCJduvQ8XqYcnp6eAPR8vT89\nX+8PwJB+r9B30DvMmj6V9atXUqJUGV7rPxiA7Dlzcj7gXILbk1DL/1hEjpw5KedXkY3r1iS4XPNW\nbWjeqg2bN6xj9Efv8cu8P6LtD7x8iUyZMzu26zVozJ6dO2jdpC7Zc+TAv0o1xzlwZc6s/7F3107m\nLLJ+P/kLPMqcRcsBOH7sCGfPnqFYiZK83uMFwsLDGPj2cIoUtW5j5MiZi4BzZx0BKCnc63kD2LB2\nNTN++oHf/3B9j/Hq1UAyZLSuvWIlHufVNwbw7JNPkC5dekqXLRfnebu77kyZM/PTL/Pseq/w1dhP\n+X76rwzo8wrXrl6h52t9qVSlGgA5cuYkIOBsrHUnl6hFEh52yTVsmwm4AiAi9URkYdQOEflKRLrZ\nP48SkQP2w62f2WnvicgA++fVIvKJWOsZ/iMite10TxH5VES22WV72ul5RWSt3QP+S0Rq23l/sLf3\niUi///rmboREsOVoIHVK5uRKUDgZfb3x9LAunjxZfDh/LQSAC9dD6T1tF23GbWDMH/84yjq7cD2U\nwwE3qeyiRxcSHklaL+sf3pWgcILCIlj6lzVMs2RPAKXzWff3nq6cn8V7rPTdJ6+S1suDrOnSuGx7\nnZI5OXDmeoxADVDqkUyIwPELt2heLg9vTN/No9nTUTCH9ak3rZcHIeGRiTtZd8mT13oWOUfOXDRp\n0Yo9O63bEHNn/UzTlm0AaNH6Kfbuctye4OD+fQzt9ypfT/uFrNmyu6zX08uL27dvJ6gNs2f+xKpl\nSxg7aWqs/+g7dOrG/BUbmTl/OZmzZOGxwkWj7d+/bzfGGAoXKc6S+b8x/tvp/HvimOP+VlhICD4+\nvglqT2Js27KJZUsWUaVscV7p3pn1a1fzWo9uCS5frWZt/j1xPNrEGAAfX19CQ6J/EHxjwGBWrN/G\nrN+XWO+1aDGXda5dvZIvPh/FDzPmkDZtzA9Xoz4YzqBh7/Hd1xN4rssLvPP+x4z55CPH/tCQEHx8\n3X+unN3reTvw1z4G9OnF1P/NJlss156XZ/Rr77kuL7B0zWZ+W7KSzFmyxnre4qt73OiPeePNwfw+\nZxZVqtXgi0nf8fmoDxz7k+O8JZQO2/43vnbQOgh8izWbKVYikh14EihtjCnHnRlSd/MyxlQB+mKt\nRwjQHWvacWWgMvCyiDwGPAcstXvA5YHdgB+Qz76ZXBbrOSBX7ekh9qoWkUExJ0ZkS5/GMXSa1suD\nGsWyc+yCtVLUliOXaVY2DwBP+edzDHlmTedN1PXQs0FhZm+zhm/yZPYhrZf1q8jk64X/Y1k5dvEm\ndzt64aYjcAH8eeACVQtbQbZGsewcOW+VOXs1hBrFrH98RXKlJ42Xh6PHereWfneGbO/Wt2kxxi09\njJen4GF/GLhtDL7eVgB/LGd6DgfEbGdCBd26xc2bNxw/r1u9kuKPW5ObcufJy5aN6wDYuG41BQsX\nsd7b6VO88sKzfDbhOx4r4vqPEEDhosX498TxeNuw5s9lfPPVWL7+6Vd8XQyFRbl08YLj+EsXzad1\nu47R9o8dNYJ+g98lIiLcMTvVw8ODEHsm6fGjRyheMubErf9q6PAP2XHgGFv3/cOk736iVp16fDXl\nhzjLHD92xHF/fu/uXYSFhcX4Y50lS1Yib0cSEmJ98IuMjCQw0Lpff+Cvffy9fx91GzSOUfe+PbsZ\n1Lc3P8yYQ46cuWLs37R+LXny5KVwkWIEBwXh4eGBeHgQ7DTj9tjRw5R8vHSizkNi3ct5O33qX17q\n3IEvv57q6CW7UqRYcU6eOObYjrp2Tp/6l8ULfudJF6Mb8dV97Ohhzp09Q43adR3nDRFCgkMceY4e\nSfrzpu5IrmHb6sA0sVZ/iM01IAT4zu6ZLowl31z7/zuAQvbPTYByItLe3s6MtfjvNuB7sRYN/t0Y\ns1tEjgGFRWQ8sAiIOesBMMZMwVqdAp88xWIMeObMlJbRHcvh4QEeIizZE8Cqvy8C8OniQ4zt5Ee/\nZsU4cOY6s7daQbJqkey82bw4Bth2LJD3f7PuRRbJlZ7BrfwxBkTguzXH+cdFUFr990WqFs7GxsPW\nH7FPFx3is2fL87avF4E3wxj8yz4ARi04yIdPl6Fbbev0RKXnypSWj9qX4eXvdwDg6+1JzWI5eGfO\n/hjHalQ6F/tOX3Pce/377HUW9q/FoXM3OHjuhuP9rLbf8724dPECr3Sz/pBERkbQ6qkO1G1g3XP8\n+PMJjBg2gMiISNL6pOWjz62ZhuM//5irVwIZPugNwOphOs/SjVKvUTO2bFxLITvovtGzK1s2rOVK\n4GVqli/KGwOH0aFTN94b3J+wsFC6Pm1NLvLzr8KHn43nfMBZhvR7le9nWJM7er/4HFevBOLl5c17\no8aSKXMWx7GWLZ5P2fIVyZ3Hmv34eJlyNK9bmZKPl+HxMtbjPps3rKF+43iXy3Srbyd/xaQvx3Dh\nfACNalaiQeNmfD5+Movm/87smdPx8vLG19eXSd9Pd/kpv279RmzdvIE69RoSHh7Ok80bAJAxYybG\nf/2DY6LV6I/ep3yFijRt0YoP3h3MrVu36NH1OQDy5S/AjzOtf7LGGMZ9NpLJU38G4Plu3en9cjci\nIyIYOWY8ABcvnMfHx5dcufMk+fmJTWznbezoj7kSGMiQN/sA4OXlxR+rY95jbNikORvXr3WMTrzU\n5RmuBF7G28ubjz/7gsxZrGtn2vfWveYuL/aIt+5PPhjOoHfeB6Bt+4680Olpvhr3KW8NsfoP4eHh\nnDh+lPIV/JPorCTSg91pdAsxrm6EuaNikZvGmAxO2+eBskBxYKgxpoWd/i2w3hjzg1hrEDYE2gOF\njDENROQ94KYx5jMRWQ0MMMZsF5EcwHZjTCERmQNMMcbEeLBVRB4BngB6A2OMMdNEJAPWYsCdgUBj\nzItxvRefPMVMgc6un89KTmm9PJjeqyodJ2zidtL82hIsjacHP79SlWcmbnbct73b0oH1krdRTi6c\nP8eA3i8zbXZsn8GST2hoKM+1bcKsBSsdAScuvmlivyeWnPbu3sU3E79k/JQEzdx3iykTviBDxkw8\n1+WF+DM/oM4HnKNPrxeZ9fuS+DO7yZIF89i3ZxcDh72X6LLN6lVnz64dbgt33jmLmGxtRrmlrgvf\nddgR1/J891Oy3PMUkZKAJ3AZOAmUEpG09uoQDe08GYDMxpjFQD+sYdaEWgq8YvcwEZHiIpJeRAoC\n540x32ANHVe0g66HMWYOMAxIulkJbhYacZsvlh12zN69n/Jm9eHTxYdiDZz3W67ceenY+QXHIgn3\n09kzp3hr2AcJCpwPknJ+FahRu65jGDo5ZMqchQ7PdU624yWF3Hny0qlrd5fPwSaViMgIer7WN9mO\np5J22NZXrNXuwerEd7UX8T0lIr9gLZl0HNhl58kIzBMRHzt//0Qc61usIdyd9pJMF7GWZaoHvCUi\n4cBNoAvWavlTRSTqg8OQe3t798f6fy7FnykZnLwUxMlLQfFnvI+eaNPufjcBgMcKF40xwSileLZz\nt2Q93jPPJ+h7iB94rZ9sH38mN2rV9sG41qM86JN93CHJgqcxJtaxJ2PMQGCgi11VXOR9z+nnek4/\nX8K+52mMuQ0MtV/OfrRfd0sxvU2llEppUkPw1BWGlFJKqURKWTdhlFJKPdBSyyIJGjyVUkq518Mf\nO3XYVimllEos7XkqpZRyH0kdE4Y0eCqllHKr1BA8ddhWKaWUSiTteSqllHKr1NDz1OCplFLKvR7+\n2KnBUymllHulhp6n3vNUSimlEkl7nkoppdxGRFcYUkoppRItNQRPHbZVSimlEkl7nkoppdwqNfQ8\nNXgqpZRyr4c/duqwrVJKKZVY2vNUSinlVjpsq5RSSiVGKvlWFR22VUoppRJJe55KKaXcRoBU0PHU\n4KmUUsqdUscKQzpsq5RSSiWS9jyVUkq5VSroeGrwVEop5V46bKuUUkqpGLTnqZRSyn0kdQzbas9T\nKaWU2wjg4SFuecV5HBEfEdkqIntEZL+IvG+nPyYiW0TkiIjMEpE0dnpae/uIvb+QU11D7PRDItI0\nIe9Tg6dSSqmUKBRoYIwpD/gBzUSkGvAJMNYYUxS4AnS383cHrtjpY+18iEgp4BmgNNAMmCginvEd\nXIOnUkoptxJxzysuxnLT3vS2XwZoAMy2038E2to/t7G3sfc3FGtmUxtgpjEm1BhzHDgCVInvPWrw\nVEop5VYi4pYXkENEtju9emeyd0EAACAASURBVNx1HE8R2Q1cAJYDR4GrxpgIO8tpIJ/9cz7gFIC9\n/xqQ3TndRZlY6YQhpZRSD6pLxphKse00xkQCfiKSBfgNKJlcDdOep1JKKfdx05BtYmbsGmOuAquA\n6kAWEYnqGOYHztg/nwEKANj7MwOXndNdlImV9jwToETeTCx7p9H9bkaKU6znjPvdhBTp6JRn73cT\nUizPeGZoqpjc/ViJtTB80v8eRCQnEG6MuSoivkBjrElAq4D2wEygKzDPLjLf3t5k7//TGGNEZD7w\nPxEZAzwCFAO2xnd8DZ5KKaVSorzAj/bMWA/gF2PMQhE5AMwUkQ+BXcB3dv7vgJ9E5AgQiDXDFmPM\nfhH5BTgARAC97eHgOGnwVEop5UbJ860qxpi9QAUX6cdwMVvWGBMCPB1LXR8BHyXm+Bo8lVJKuZWu\nMKSUUkqpGLTnqZRSyq1Sw7eqaPBUSinlProwvFJKKaVc0Z6nUkopt0mu5zzvNw2eSiml3CoVxE4d\ntlVKKaUSS3ueSiml3EqHbZVSSqlESgWxU4dtlVJKqcTSnqdSSin3ER22VUoppRLFelTlfrci6emw\nrVJKKZVI2vNUSinlRsnzlWT3mwZPpZRSbpUKYqcO2yqllFKJpT1PpZRSbqXDtkoppVRi6FeSKaWU\nUsoV7XkqpZRyG/1KMqWUUuoepIbgqcO2SimlVCJpz1MppZRbpYKOpwZPpZRS7qXDtkoppZSKQXue\nSiml3CeVPOepwVMppZTbiC4Mr5RSSiVeKoides9TKaWUSiwNnm4WEhJCs/o1aFDTnzpVyzP64/cd\n+4wxjBzxDjUqlqJ25bJ8O/krAK5fu0bnjm0dZWZM/9Fl3cHBwbRt0ZDIyEgARrwzmDpVy1O7clne\nHtgPY0yMMu8PG0ytSmWoX6MiL3Rqz7WrVwHYunkj9WtUpEndahw7ehiAa1ev0rFtC27fvu0o/3Tr\nZly9csU9JwdI6+3Bqg9asGFUS7Z82pqh7cs79k3qVYO9XzzJ+pEtWT+yJWULZo1WtmLh7AROf542\nVR4FoGzBrKx4vzlbPm3Nxk9a8VS1QrEed1SXStQomQuAP4Y3dRzj0MT2/K9/PQAy+Xoza0B9R9s6\n1S0So54MPl6OsutHtuT4lA6M6lIJgJ5NS7J5dCtmD2yAt6f1T6taiVyM7FzJUT57xrTMHdww8Scu\nFiEhITStV4N6NfypXaU8n3x053rr27sH9Wr4U7d6RV7s3JGbN28CMOmrcdSqXI661SvSrlVTTv17\n0mXdwcHBtGluXW/r166mfs1KjleBnBlZvHBerO1aMG8uuTKlYffOHQBs2byRutUr0rhuNY4duXO9\nPd0m+vXWzs3XW2zOnD5Fm+aNqO5fjhqVyvP1hC8d+z75aASlixWkbnV/6lb3Z/nSJQCEh4fzao8X\nqFXFj2oVyzL2s09c1m2MoU2Lxly/fh0Av1JFqVXFj7rV/WlQu6rLMuPHfe44Xs3KfuTMlJYrgYFc\nuniRFo3rUrOyH4sW3DnfnTo+xblzZx3b7w4dyNrVq/7zeXEXDxG3vB5kGjzdLG3atMxZsIw/N+xg\n5frtrFqxjB3btgAw8+dpnDlzmvXb/2Ldtn20adcBgKnfTKJ4icf5c8MO5i5awftvDyQsLCxG3TOm\n/8ATrdri6enJti2b2LZlE6s27mT15t3s3rmdjevXxihTt35DVm/ezaqNOylcpBhfjrH+wU8eP5af\nf53PB6M+58fvvwFg7Kcj6fPmIDw87lwW7Z95jh++ney28xMafpuWHy6j5uCF1By8gEblH6Fy0RyO\n/e/8vINaQxZSa8hC9p2880fUQ4T3n6vIn3vv/MEIDo2g56T1VH1rPk+NWsGoLpXInM47xjGzZUhL\n5aI52XjwAgDN3l/qOMbWfy6yYNu/ALzcpAQHz1yj5uCFtBixlI+fr+QIglFuhkQ4ytYaspB/L95i\n/larfIeaj1F90AK2HL5Iw/KPADDoyXKMnrvXUf7yjVACrgRTtXjO/3oqAft6W7iM1Rt38OcG63rb\nvtW63j4Y+RmrN+5gzaad5Mv/KN9PmQhA2XJ+LFuzmTWbdtKyzVOMeHeIy7r/99Od661WnXqs2rCd\nVRu2M3fBMnx901GvQWOX5W7euME3k77Cv1IVR9qk8WOZMdu63n6wr7cxn46k74Do19vTHZ9jqhuv\nt9h4enkxYuRoNu3Yy9JV6/num8kc/PuAY/8rr73Bmk07WLNpB42bNgdg3m+zCQsNY/3W3fy5fgs/\nfv8N/548EaPu5UuXUKZsOTJlyuRIm7d4BWs27eDPdVtctuf1vm86jvfO+x9So1YdsmbLxpxfZ9Kt\new+Wr9noCPB/LF5IuXJ+5M37iKP8y71688WY0e44NW4h4p7Xg0yDp5uJCOkzZACsT6oR4eGOm+c/\nfvc1bw582/HHImfOXI4yN2/exBjDrZs3yZI1G15eMW9Hz/1lBk1btHKUCQ0JISwsjNDQUMLDw8mZ\nK1eMMvUaNnbU5V+5KufOngHAy9ub4OAggoOC8Pby4sSxo5w9c4qatetGK9+0eSt+mzPLHafG4VZo\nBADenh54eXrgosMcQ69mJZm/5V8uXg9xpB0JuMHRgBsABFwJ5uL1EHJk8olRtnWVR1mx52yM9Iy+\n3tQpnYeF208BYOw0gAw+3ly5GUqEU6/obkXzZCRnZh9HUBax3pNvGi8iIm/zTK3CLN9zhiu3on8Q\nWrj9FB1rFY7/TSeAiJDB6XoLj7hzvWW0/3gbYwgJCXb8NapVpx7p0qUDoFLlKpw9c8Zl3XN+mUGz\nJ1rFSF8wby4NGjd11HG3UR++x+t9B5DW587vwtvbm+Ag+3rz9uJ4LNdbsxat+G22e683V/LkyUt5\nv4oAZMyYkWIlSkbrybkiCEFBt4iIiCAkOJg0adKQMWOmGPlmz5pB8yda33Pb5v46i3ZPdwTunLew\n0FA8PD2JiIhg8oQveb3fgGhlCjxakCuBlzl/PuCej6sSJ1mDp4i8LSL7RWSviOwWEZdjGCJSSUS+\ndNr2FpHjdpndIhIgImecttMk8Pj1RGShu95PbCIjI2lYqxJliuajTv2GVLQ/gZ88fox5c3+lSd1q\nPNuulWO49MUer3L4n4OUL1GQ+jUq8sEnn0f7NA4QFhbGyRPHebRgIQAqValGjdr1KF/iUcqXeJT6\nDRtTvMTjcbZrxvQfaNC4KQB9+g/k9Z4v8uXY0bzY41VGfvAug995P0aZLFmzEhYaRmDg5f96Whw8\nRFg/siVHv+7Aqn3n2H70kmPfux0rsPGTVozsXIk0XtY5yJvVl5aVC/DtikOx1ulfJDtpvDw4dv5G\njH3VSuRi1/GY7W9ZqQBr9gdwIzgcgClLD1L8kcz8M7E9m0a3YtC0bXEG9nY1HmPuphOO7SlLD7Hy\ngxYUyJGezYcu0KleEaYsOxij3K5jl6heIuYHnXsVGRlJ/ZqVKFUkH3XrN8S/8p0eX59XXqJ00QIc\n+ecQL/XsHaPsz9N+oKF9TTi7+3pz9vucX3iqfUeXbdm7exdnzpyicbMW0dL79B/Iaz1f5Msxo+ne\n41VGjniXIcNcX2+hoWEEXnbf9Raff0+eYN+e3dF6yt9+PZHaVSvw+isvOYaRWz/ZjnTp0lOqSAHK\nP16Y3n36kTVbthj1bdm8kfIVKjq2RYT2bZrToFYVxyhPbIKCgli5Yimt2jwFQPsOz7Jk0QKeat2M\nfgMG892USXR4tpPLDy7l/CqwZdPGezoH7mT1GsUtrwdZsgVPEakOtAQqGmPKAY2AU67yGmO2G2P6\nOCXVAhYaY/yMMX7AZGBs1LYxJuYY533k6enJyvXb2XXgOLt2bufvA38BEBoWSlofH5at2czzXV+k\nX+8eAKxauYwyZcuz59BJVq7bxtABfblh3y+JEnj5EpkzZ3ZsHz96hMP/HGTXgePs/vsE69euZvPG\n9bG2adynI/Hy8qJdh+cAKFPOj8Ur1zN34XJOnjhO7jx5McbQo9tz9H65KxcvnHeUzZEzJ+fj+VSe\nGLeNodaQhTzeezb+RXLweP4sALw3cxf+b86j3tuLyJohLf1alwFgVJfKDP/fzlgDWe4svkx5tRav\nTt7oMk+eLL5cduqxRmlf4zFmbzzu2G5Y7hH2nQyk+KuzqTV4IZ92q+LoibrSrnqhaOVnrj9G7SEL\neXnCenq3KMXkPw7S2C8f0/rWZWTnSo5hqIvXQ8ib1XWv7V54enqyasN29vx9nF077lxvAF9O+pZ9\n/5ykWPGSzJv7a7Ryv878mT27dtD7jTdj1Hn39RblfMA5/t7/F/UbNYmx7/bt27w79C3e/yjm8GHZ\ncn4s+XM9vy2Kfr293O05XnmpKxfuut4CAtx3vcXl5s2bdOvUgY8++dwxzPrCSz3Zse8QazbtIHfu\nvLwz9C0Adm7fiqenB/uP/MvOvw4zYfw4Thw/FqPOK1cCyZgxo2N70fLVrNqwjVlzF/LdlElsXL8u\n1vYsXbyQqtVqOIJypsyZmTlnPn+u20J5vwosXbKI1m3b0fe1nnTr1JFtWzY5yubIkYsAN/47/S88\nxD2vB1ly9jzzApeMMaEAxphLxpizIlJZRDaKyB4R2SoiGV30EJsBS1xVKiL+IrJGRHaIyFIRyWun\nFxWRFXa9O0UkavZHBhGZLSIHReRnScKPN5mzZKFm7bqsWrEMgEceyUeLVm0BaNGqLQf27wOse6Et\nWrVFRHisSFEeLViIw4ej97J8fHwJCQ11bC9eOA//ylVInyED6TNkoEHjpmzfutllO2b+PI3lSxcz\n4ZtpMT7NGWMY9+lI+r01lM9Hfcg7I0bSqWt3x2QmgNCQEHx8fP/7CbnLtaBw1h0IoJF9f/D81WAA\nwiJuM331EfyLWPdCKxTOzvd96rDvy6doU7UgY16syhOVCgDWMOuvAxswYtYuth255PI4wWGRpPX2\njJaWLWNa/IvkYOmu04605+sVddy/PHb+Bicv3qT4IzGH5QDKPJoVL08Pdh8PjLEvT1Zf/IvkYNH2\nU7z+RCm6fbGWa0Fh1CuTFwAfb0+CwyISfJ4SKup6+9O+3qJ4enryZPsOLJz3myNtzaqVjPtsFNNm\nzSVt2rQx6vLx8SXU6XqLMm/ubFq0aoO3d8wPFTdv3ODggf08+URj/MsUY8e2LXR+5inHpCGwrrcx\no0fSf+BQPh31Ie+OGEnnbnddb6FJc73dLTw8nG6dOtC+47O0avOkIz1X7tx4enri4eFBlxe6s3P7\ndgBm/zKTBo2b4u3tTc5cuaharXq09xbFy8sr2iSoRx7JB0DOXLl4olVbdu7YFmub5s7+haeedt2r\n/+yTj+j/1hDm/jqTqtVrMmHK93zy8QeO/aGhIfj6Jv15U5bkDJ7LgAIi8o+ITBSRuvZw6yzgDWNM\neazeaLCLsvWB1Xcniog3MB5ob4zxB74HPrJ3/wxMsOutAZyz0ysAfYFSQGGgpqvGikgPEdkuItsD\nL7v+o+zKpUsXHTNag4ODWbtqJUWLlwCg2ROt2bBuDQAb16+lcJFiAOTLX4B1a/4E4OKF8xw98g8F\nCz0Wrd4sWbNyOzKSkJAQR5lN69cRERFBeHg4m9avo3iJkjHa8+eKpUz44jN+nDnX5VDPLzN+omGT\nZmTNlo3g4CA8PDzw8PAgONj6NRhjuHDhPAVcDN/di+wZ0zom9fh4e1K/bF4On70GWD3IKC0rF+DA\nKes8lnvjN8r2mUvZPnOZt+Uk/b/fwqLtp/D29ODn/vWYue4Y8+yg58qhM9conCdjtLS2VQvyx67T\nhIbf+SN36tItR4DLmdmHYnkzc/zCTZd13t1rdTbsaT8++nU3AL7eXhgMt28bfNNY956L5s3E36ev\nxn6SEuHu623NqpUUK1YCYwzHjh4BrN/hH4sXOq7DfXt2MeCN3vw0c67jvvvdsmTNSqTT9Rblt9mz\neDKWIdtMmTNz8MQ5dvx1mB1/Hca/clV+mjkXv4r+jjyz/vcTjVxdb0FO19v58y6Hi93JGEOfV1+m\neImSvPp6v2j7AgLOOX5etOB3Hi9VGoD8BR5l3RprRuutW7fYvnUrxUqUiFF30WLFHT3SW7ducePG\nDcfPq/5c7qjvbtevXWPjhrUu75cePXKYs2fOUKtOXYKDrPMmIoQEB0fLUzKWupNbahi2TbZFEowx\nN0XEH6iNFQxnYQW6c8aYbXae6xB9UWERyQcEGmOCXFRbAigDLLfLeALnRCQjkM8Y85tdb4hTvVuN\nMaft7d1AISDGeKcxZgowBaB8Bf8ETGmxXAg4R59e3Ym8Hcnt27dp/WR7mjR7AoDX+w3k1Ze7MmXi\nF6RPn4Ex461Zhf0HDuWNV16iXvUKGGMY9v5HZM+eI0bddes3YuumDdSp35BWbduxYe1q6levACI0\naNSUJs1bWvW91pMuL/bAr6I/Qwf0JSwslI5trRmD/pWqMnrcBMC6vzLrfz8x67fFAPTs3ZdOT7cm\njXcaJn43DYA9u3biX6mKywlM9yJPVl8mv1ILTw/BQ+C3zSf5Y5c1YeXb12qRI6MPIrDv5BX6fuu6\nJx3lqeoFqVkyN9kypOW5OtbAwiuTN0SbpQuwdNdpXmxUnGmrjjjS2lUvxNj5f0XLN/q3vUzuVZNN\nn7RCBIbP2EHgDav3tX5kS2oNuTMY8mS1grQfvTJGm8oVsobb9pyweqS/bjzO5tGtOXP5FuMW7Aeg\ndqk8LN3lepJOYp0POMfrvboTGRmJibremj/B7du3eb1Xd27euI4xhlJlyvHpWKt39947Q7h16ybd\nuz4LQP78Bfhp1m8x6q7XoBFbNm2gbn3r0Zp/T57gzJnT1KhVJ1q+UR++h19Ff5q1iDm5yFnU9fbL\n79b19sprfXmufWu806RhsvP1Vtl911tstmzawC8zfqZU6TLUrW4F92HvfUjjps15b9hg/tq7BxHh\n0YKF+PxLa5Zy9x6v8Hqvl6hRqTzGGJ7r3JXSZcrFqLtJ0xZsWLeGwkWKcvHCebo82x6AiIhI2nV4\nxnGPeeq3XwPWMDHAwgW/U79BY9KnTx+jzo/ef5e3h48A4Kmnn6Hzs+344vNPGTxsOGD1oo8dPUqF\nipVilL0fHvC45xbi6tnAZDmwSHugN5DGGFPzrn31gAHGmJYi0h3IZIwZ67T/PeAmsBSYYoypflf5\njMDfxpj8sdVrb38FbDfG/BBXW8tX8DfL1sT9hzw57N29iykTv+CrKT8k2zGHDepP0+YtqV2vQaLL\nFus5IwladG+WDm9Gh09Xci0o/H43hSXvNuXZz1dx9ZbrW/VHpzybzC1ybe/uXUye8AUTv/kh2Y75\n9sD+NG3Rkjr3cL0BeD4AN8oCAs7x6ssvMHfBH8l2zIXzf2fv7l0MfTfmJKz4NKhdld07d7jtxGUu\n+LipNdT1s+qJtbhX1R3GGJefCESkADANyI01WX6KMeYLOz68DFy0sw41xiy2ywwBugORQB9jzFI7\nvRnwBVYH7FtjzKj42pacE4ZKiEgxpyQ/4G8gr4hUtvNkFJG7P3LGer8TOATktCcjRc3KLW2MuQGc\nFpG2dnpaEXHfDI37pJxfBWrUrudYJCE5lHy89D0FzgfN29O3kz9HzE/0yS17xrR8tfhArIHzQVLO\nrwK16iTz9Vaq9D0HzgdFnjx56dKtu2ORhOQQGRFB7z794s+YDAR7fVs3/BePCOBNY0wpoBrQW0RK\n2fucJ5RGBc5SwDNAaay4MlFEPEXEE5gANMe6nfesUz2xSs61bTMA40UkC9abPgL0AKba6b5Y9zsb\nRRWw31RRY0zM+f6AMSbM7sF+KSKZsd7POGA/0Bn4WkRGAOHA00n2zpLRc527Jevxnu/WPVmPl1Sc\nH4e5ny7fCGXRdpeTzB9IyX29dX5Irre27ZL3z02bp9on6/HikxwDAMaYc9hzWYwxN0TkbyBfHEXa\nADPtSavHReQIEPV80hFjzDEAEZlp5z3guhpLct7z3IE1cedul7A+NThbDawWkVpAjCU5jDHvOf28\nG6jjIs9h4O6PsMdwmnhkjHktQY1XSil1P+QQke1O21Ps+SjRiEghrMmgW7Amgb4mIl2A7Vi90ytY\ngdX5/ttp7gTbU3elu15H0ckD/a0qxpj1uJjMo5RS6gHl3pmyl2K753nncJIBmAP0NcZcF5FJwAdY\n90E/AD4HXnRXg6I80MFTKaVUypNcs23txxXnAD8bY+YCGGPOO+3/BoiaJn8GKOBUPL+dRhzpsdK1\nbZVSSqU49gI332E9WTHGKT2vU7Yngahn0uYDz9gTSB8DigFbgW1AMRF5zF574Bk7b5y056mUUspt\nBJLr68RqYk0M3Wc/sw8wFGu2rB/WsO0JoCeAMWa/iPyCNREoAuhtjIkEEJHXsB599AS+N8bsj+/g\nGjyVUkq5VXLETntOjKsjLY6jzEfcWYXOOX1xXOVc0WFbpZRSKpG056mUUsqtHvR1ad1Bg6dSSim3\nsb7P8363IunpsK1SSimVSNrzVEop5VbJNNv2vtLgqZRSyq0e/tCpw7ZKKaVUosXa8xSR8VgPmbpk\njOmTJC1SSimVoqX22bbb49inlFJKxWCtMHS/W5H0Yg2exphoXwUuIumMMUFJ3ySllFLqwRbvPU8R\nqS4iB4CD9nZ5EZmY5C1TSimV8thfSeaO14MsIROGxgFNgcsAxpg9uPjyaaWUUgruLJTwX18PsgTN\ntjXGnLorKTIJ2qKUUkqlCAl5zvOUiNQAjP3Fo28Afydts5RSSqVUD/qQqzskJHj2Ar4A8gFnsb7z\nrHdSNkoppVTKlOpn20YxxlwCOiVDW5RSSqkUISGzbQuLyAIRuSgiF0RknogUTo7GKaWUSnl0tq3l\nf8AvQF7gEeBXYEZSNkoppVTKJW56PcgSEjzTGWN+MsZE2K/pgE9SN0wppZR6UMW1tm02+8clIjIY\nmIm11m1HYHEytE0ppVQKI6JfSbYDK1hGnYWeTvsMMCSpGqWUUirlSgWxM861bR9LzoYopZRSKUWC\nvgxbRMoApXC612mMmZZUjVJKKZVyPegzZd0h3uApIsOBeljBczHQHFgPaPBUSikVQyqInQmabdse\naAgEGGNeAMoDmZO0VUoppdQDLCHDtsHGmNsiEiEimYALQIEkbpdSSqkUSJBUP9s2ynYRyQJ8gzUD\n9yawKUlbpZRSKmVKAV8n5g4JWdv2VfvHySLyB5DJGLM3aZullFJKPbjiWiShYlz7jDE7k6ZJDx4v\nDyFzOu/73YwUJ+DHzve7CSlS9qqv3+8mpFgXN395v5uQ4iTFEGtqn237eRz7DNDAzW1RSin1EEjI\nTNSULq5FEuonZ0OUUkqplCJBiyQopZRSCSHosK1SSimVaB4Pf+zU4KmUUsq9UkPwjPe+rlieF5F3\n7e1HRaRK0jdNKaWUejAlZFLURKA68Ky9fQOYkGQtUkoplWKJWPc83fF6kCVk2LaqMaaiiOwCMMZc\nEZE0SdwupZRSKZQO21rCRcQT69lORCQncDtJW6WUUkrFQUQKiMgqETkgIvtF5A07PZuILBeRw/b/\ns9rpIiJfisgREdnrvBCQiHS18x8Wka4JOX5CgueXwG9ALhH5COvryD5O9DtVSimVKoi45xWPCOBN\nY0wpoBrQW0RKAYOBlcaYYsBKexusr9MsZr96AJOstko2YDhQFagCDI8KuHFJyNq2P4vIDqyvJROg\nrTHm73jfllJKqVRHSJol/+5mjDkHnLN/viEifwP5gDZY30EN8COwGhhkp08zxhhgs4hkEZG8dt7l\nxphAABFZDjQDZsR1/IR8GfajQBCwwDnNGPNvgt+lUkoplXg5RGS70/YUY8yUuzOJSCGgArAFyG0H\nVoAAILf9cz7glFOx03ZabOlxSsiEoUVY9zsF8AEeAw4BpRNQVimlVCrjxrVtLxljKsWVQUQyAHOA\nvsaY686zdI0xRkSM+5pzR0KGbcs6b9s3WV+NJbtSSqlULrmeMhERb6zA+bMxZq6dfF5E8hpjztnD\nshfs9DNAAafi+e20M9wZ5o1KXx3fsRP9AcH+KrKqiS2nlFJKuYtYXczvgL+NMWOcds0HombMdgXm\nOaV3sWfdVgOu2cO7S4EmIpLVnijUxE6LU0LuefZ32vQAKgJn4yunlFIq9RGRZJkwBNQEOgP7RGS3\nnTYUGAX8IiLdgZNAB3vfYqAFcARrHs8LAMaYQBH5ANhm5xsRNXkoLgm555nR6ecIrHugcxJQTiml\nVCqUHLHTGLMeay6OKw1d5DdA71jq+h74PjHHjzN42osjZDTGDEhMpUoppdTDLNbgKSJexpgIEamZ\nnA1SSimVsqWG5fni6nluxbq/uVtE5gO/AreidjrNbFJKKaWA5Fsk4X5LyD1PH+Ay0IA7z3saQIOn\nUkqpVCmu4JnLnmn7F3eCZpQkeehUKaVUypcKOp5xBk9PIAOuZzNp8FRKKRWT6D3Pc8aYEcnWEqWU\nUiqFiCt4poLPDkoppdxNUkH4iCt4xnjIVCmllIqLNdv2frci6cW6tm1ClidSSimlUqOEPKqilFJK\nJVhq6Hlq8FRKKeVWkgqeVXHjd5YqpZRSqYP2PJVSSrlNapkwpMFTKaWU+0jqWGFIh22T2NWrV3m2\nY3vKlymJX9nH2bxpEwDvD3+HyhXKUdXfj5bNm3D2rOvvF9+9axe9Xu4OwKGDB6lbqzqZ06dl7JjP\n4j12/759yJElg2N74lfj8fcrQ9tWLQgLCwNgw/r1vPVmP0eeixcv0vqJZvf8fv+LXj1epGD+3FSq\nUDZaemBgIC2bN6FcqeK0bN6EK1euALB2zWry5sxCtcoVqFa5AiM/cr2mhzGG5k0bcv36dU6fOkXz\nJg3wL1+aSn5lmDD+C5dlFs6fRxX/8lSrXIFa1SuzccN6AP45dIia1SpRxb88WzZbv8uIiAieaNaY\noKAgR/muzz/LkcOH//M5uZuHh7BpxiDmfNHLkdarY53/t3ffcTJdbQDHf88WrN7ZBNFLlOidCKJF\n7733kIREIgkhDUEQaC50QwAAIABJREFUIryIdETvEaK31VskwWqJFr3uWnb3vH/cu2PWzK7dzWCX\n5+szn505c8+5xZ157ilzD78vGkLwni9JlzqZIz1l8iTMHdeDbT8PZNfc92lXv6zjvUVf9ubshpGR\nynFn1FtNqFA8FwAvlsrLlhnvsHPOe0z9qB3e3tbXR8vaJdn+87vsmP0ea7/tT+G8z7otK6r8DasV\nZdfc9/nt6zdIm8ra/hxZ0vPDiE6OvL4+3qz6+g1HHk/o1b0LObJmpnTxIpHSh338IXlzZqV86eKU\nL12cX1csB+DSpUvUqVGNzOlS8uYbfaMtu22rZhw/dgyARvVqU65UMUoVK8zrfXoRFhYWZb5dO3eQ\nOlkiFs6fC8Dhw4eoVK4UZUsWjXS+1atdI9L51rFdKwIDPX++qahp8HzI3ur3OjVq1GLf73+xfdc+\n8hcoAEC/NwewY89+tu3aS+06dRn+ifsv/pGfDaN3n9cASJM2LZ+PHc8b/R88vequnTu5ageZCLNm\n/sSO3fspW648q1b+ijGGEcM+5t33BzuWyZAhA5kz+7Nl8+a47nKctW3XkYVLfnFJ/3zUCKpUrcr+\nPw5TpWpVPh81wvFe+QqVCNixh4Ade3j3/Q/clvvrL8spXLgIKVOmxNvHh2GfjWbXvoOs3biVKZO/\n4s8//3DJU6VqNbbt3EvAjj1MmvI1r/bsBsDX0/7HqM/HsWDRMr4Y+zkAU/83iZat25A0aVJH/q7d\nezJ2zMj/dDzc6dP6JQ4d/zdS2ta9x6jTcwInz1yKlN6jeWX+OnaOMi1GULPbF4zo3whfH28Axn7/\nG10GfR/tutKmSkbpwtnZvPsoIsK0j9rRfuA3lGw2jL/PXqZtvTIAnDhziRpdx1Gq+TCGT13BxEGt\nXMqKLn+vli9Sse1Ips3bTIvaJQEY+mpdhn611JH/bmgYa7cdolmN4rE8YlFr064DCxYvd/veq33f\nYMv23WzZvpuateoAkCRJEgYN+ZBPR0T///rnHwcJCwsjR86cAHz3089s3bGH7bv3c/HiBRbMm+M2\nX1hYGB+8/y7Vqr/sSJs+bQqfjR7LvIVLGT/OOt+mTZlMi1atI59v3Xoy7vNRMd/5h8xLxCOP+EyD\n50N07do1Nm3aQMfOVs0xUaJEpE6dGoCUKVM6lgsKuuV2dNqNGzf4/cB+irzwAgAZM2akZKlS+Pr6\nRrvesLAw3hs4wOVDbozh7t27BAUF4evry8yffqRGzdqkTZs20nL1GjTk55k/xX6H/6OKlSqTNk1a\nl/RlSxbTpm0HANq07cDSxYtiVe6sWTOoW68BAP7+/hQrZn0Bp0iRgnz5C3Dm9GmXPMmTJ3f8nwTd\nuvf/4+vrS1BQkOMYXr16leXLltKmbftI+StUrMTa1asJDQ2N1bZG59mMqalVsSDfLNgSKX3foVP8\nfdb1Z9kGSJ4sMQDJ/BJz5VoQoWHhAKzbfpgbt0KiXV/DakVZueVPANKlTsadu6EE/n0egDUBf9Gw\nWlEAAvYd5+qNYAC27z/Os5lSu5QVXf7w8HAS+/qQNEki7oaGUaFYLv69eJ2jf1+IVMaSdftpUadU\ntNscGxUrVSaNm/MtKsmSJaN8hYokTpwk2uV+njmDV+rWd7yO+KyHhoZy586dKEeiTv7qSxo0akz6\nDBkdab6+vgQHBREUfO98+2XZElrfd76Vr1iJdWs8e77FVUSfpyce8ZkGz4foxPHjpE+fge5dOlG2\nZDF6de/KrVuOKVEZMvh9cufIyqyZPzF4qGvNc/eunTxfsFCs1ztp4pe8Urc+/v7+kdJ79e7DixXK\n8s/ff1OufAW+/+4bevZ+1SV/8RIl2bxpY6zX+7CcP/+vY18yZ87M+fP3al7bt22lTMmiNKxXhz/+\nOOg2f8DWzRQrXsIl/eSJE+zbt4dSpcu4zbd40QKKFS5Ak4Z1mTTlawB69HyV0SOH071rR956511G\nDPuYAe+8i5dX5I+Sl5cXOXPl5sD+fXHaZ3dGDWjC+18sJDw8ZvMyTJ61nvw5MnNs5afsnPMeb42a\nizExn9OhXNGc7PnzbwAuXrmJj483xZ/PBkCj6kXJkimNS56ODcvz62bXmnx0+UdNX8WyyX2pU7kQ\ns1fsZGC3WgyfusKljIOBZyhRMFuMt/+/mDJpImVLFqVX9y6OboKYCti6xeV8a1i3FjmzZiZF8hQ0\nbNzUJc+Z06dZsmghXbtHbkbv3qM3o0eOoEfXTrz19rt8NvwT3no7qvMtl0fPNxW9eBU8ReR9ETko\nIvtFZK+IuP9Wi12ZVUSkvCe2L7ZCQ0PZu2c33Xr0ImDnHpImS8bokfeaHD/8+FMCj/9Dy1ZtmPzV\nly75z549S4b0GWK1zjNnzjB/3hx693Htk2ndth0BO/fwzfc/MuGLsfTu8xq/rviFVi2aMuDNfoSH\nW7WSjBkzcvas+z7Yx01EHFfuRYsV588jJ9i2cy89e/ehZdNGbvNcuXyZFClSREq7efMmrVs2ZeTo\nsZFaAZzVb9CIPQf+ZNacBXw01GoSzpotGytWrWXthi0k9UvKmdOnyZe/AF06tad9m5YcOXzYkT9D\nxoycjaIvO7ZqVyrE+cs32PPnPzHO83L5Auw/dIqcNd6nTMvhjB3YjBTJoq81OcucPiUXr9x0vG4/\n8BtGvtmYjT+8xY1bIYTZ50uEyiXz0KFhOQZ94b5lIKr8a7b9RYU2I2n6xv+oW6UIv246SJ7nMjJj\nVBcmDm6FXxKrpSU83HD3bhjJkyaO8T7ERdfuPdn/5xG2bN9N5sz+vPfOg7tJnJ07d5b0931uFy5d\nwZETpwm5E8L6tWtc8rwzoB8ffTrcJShmzZaNX1atYc36zfglTcqZ06fIl78A3Tq1p0Pblhw54nS+\nZYg/n1sRzzzis3gTPEWkHFAXKG6MKQJUB2L+TeG+TB+gCvBYguezWbLwbJYslC5jXQM0atKUvXt2\nuyzXolUbFi6Y55Lu5+fH7du3Y7XOfXv3cOxoIAXz5yZf7uwEBQVRMH/uSMucOXOGnTu2U79BQ74Y\n+zk/zviZ1KlTs3bNagBu375NEj+/WK33YcqYMRNnz54F7AsKu1krZcqUJE9uDYiqVbsOd0PvcvHi\nRZf8Pj4+jgsDgLt379K6RVNatGxNg4aNH7j+ipUqc+L4MZeyPxwyiA+GfsykiePp2KkLnwz7jGFO\ng5ZCPHgcyxXNSd0XC/PXsg/5fkQnqpTKy/RP2kebp139sixaY9VEjv1zkROnL5Eve6YYrzM45C6J\nE93rIti2/zjVu4yjUrvRbNodSODJ8473CuV5hkkftKZZvylcvnbLXXHR5gfwS+JLu3plmDx7A4N6\nvkLXwT+wZe8xWta+11SbyNeH23fuxngf4iJjpkx4e3vj5eVFx85d2bVzR6zyR/W5TZIkCa/Urc+y\npYtd3tuzaxed2rWmYN6cLFowj36v92HJ4oWRlvloyCAGD/2YSRMn0KFTFz7+9DNGOI2VuB1yG78k\n8eFzK3h56BGfxZvgCfgDF40xIQDGmIvGmDMickJERorIARHZLiK5AUQku4issWupq0Ukm53+rYhM\nFpFtwGygJ9DPrslWEpFmIvK7iOwTkQ0Pc4cyZ85MlixZOXzoEADr1qwmf4HnASKNxFy6eBF58+V3\nyZ8/fwGOHg2M1Tpr13mFE6fOcSjwBIcCT5A0aVIO/hW5jI+GDGbwEOtDFxwcjIjg5eXlGL135PBh\nCsahufhhqVO3Hj/9+B0AP/34Ha/Us/qTzp0752iG3LljO+Hh4aRLl84lf568+RwjH40x9OrRlXz5\n8/PaG/2jXOfRwEBH2Xv27CbkTkiksjduWE9mf39y58lDUFAQXl5eeHl5Eew0AvLIkcNxanZ354MJ\ni8ldazD5XxlC+4HfsG7HYTo/YMDPP+euUKV0PgAypk1B3uyZOH7a9eIiKoeOnyNXtvSO1xnSWBcq\niXx9eLPjy0yda41Azpo5DbNGd6PL4O8dfZruRJU/Qr/21flq5npCQ8PxS+KLwRAeHk7SJIkAawDT\npas3CQ0Ndynbk87ZF2oASxYv5PmCBWOVP1/+/ByzP7c3b950lBcaGsqvK5a7/az/fugoBw8f4+Dh\nYzRo1ISxX3xJvfoNHe9v2rAef/9nyJ07D8HBQYh9vgUF3zvfAo8c8dj5ph4sPv3OcyXwgYgcBn4D\nfjbGrLffu2aMKSwi7YFxWDXUCcB3xpjvRKQzMB6IONuyAOWNMWEiMhS4aYwZDSAiB4CaxpjTIuI6\nssEmIt2B7mA1ncTVmHET6NS+DXfu3CF7zpxMmfYNAIPeH8iRw4fwEi+yPfcc4ydOdsmbL39+rl+7\nxo0bN0iRIgXnzp2jQtmS3Lh+HS8vL74cP449+/8gZcqUNKxXh6/+N41nnnkm2u3Zu2cPAMWKW4Nm\nWrRsTclihcmSJSv933obgPXr11Kr9itx3ue46tCuNRs3rOPSxYvkyZmVQYOH0qFTF94cMJB2rVvw\n/TfTyZrtOX6Y8TMAC+fPZdqUyXj7+ODn58d3P8x0OxijVu06bNywjly5c7N1y2Zm/vQDBQsVpmyp\nYgAM/ehTatWuw7Qp1v9B1+49WbhwHjN//AEfX1/8/Pz4/sdZjrKNMXw2/FO+/2kWAJ27dKdzx7aE\nhobyxYSvAPj333/x8/Mjc+bMD/249W71Iv07VCdTupTsmP0eKzYdpPdHMxgxdQVTPmzLjtnvIQLv\nf7GIS1etWuFvX79B3hyZSO6XmMAVH9Pzwxn8tvXPSOWu2HiQLk0q8O0C6ycS/TpUp3alQnh5CVPn\nbGT9DqvJ8N3utUmbOhnj3m0BQGhYOBXbWIPVFkzoRe+PZnD2wrUo8wP4Z0hFyULPMWyKNdp60sz1\nbPrxba7dCKJ5/6kAvFgqDys2ue/XjotO7VqzceN6Ll28SL5c2Xhv0BA6dOrC4PfeYf/+fYiI9dn8\n8t5ns2DenNy4cZ07d+6wdMkiFi1d4bggjlCzVh02bljPS9WqE3TrFi2aNiQkJITw8HAqv1iFLt16\nAPD1VKvcLt2i/7mQMYaRIz7l2x+t861jl2507diO0NBQxo6fCMB5+3zL9AjOtwcR4n+TqydIbAYQ\nPGwi4g1UAl4CegADgaFAVWPMMRHxBc4ZY9KJyEXA3xhz104/a4xJLyLfAmuNMd/ZZQ4lcvCcDOTC\nqpXON8ZEHt/vRokSJc3mbTs9vLcxM37cWFKkSEGnLl0f2Tqrv1SZOfMXkSaN64CQ2IjpwJaH7ezZ\ns3Tr3IGlv6x8ZOuc8IXVl9qhU5dY501XJvrfED5Kq6f3o/Frk7l2M/hxbwqzRndl0PjF0dZuLwSM\nf4Rb5F5wcDB1albjt7Ub8fb2fiTr/HL8OFKkSBGn861y+dLs3rXTY+HuuQJFzLvTXZum46JX+Ry7\njDElPVKYh8WnZluMMWHGmHXGmCFAH6BJxFvOi8WgKPedLtY6egKDgKzALhFxbeeLR7r37EXixA93\ngISzCxcu8Nob/f9z4IxP/P396dSlK9evX39k60yVOjVt2nV4ZOt7WAaOmU9W/8d/Lvj6eLN43f5o\nA2d84efnx/uDh7j9CdTDkipVqififEtI4k2zrYjkA8KNMRGdgUWBk0BhoAUwwv671X5/C9AS+AFo\nA0T124obgGM4pYjkMsZsA7aJSG2sIPrA2ufjkiRJElq3bffI1pchQwbqN2j44AUTmCZNmz/S9bXv\n0OnBCyUAO34/+bg3AbBukjBj6fbHvRkxVv3lmo90fe3i2fkW329w4AnxJngCyYEJdj9kKBCI1edY\nF0gjIvuBECDi9iV9gW9EZABwAYjq7FkCzBWRBnaefiKSB6tpfjWgP4xSSikPeVr6PONN8DTG7MLN\nT0rsQRqjjDHv3Lf8SaCqm3I63vf6MOB888r48+t/pZRSCVK8CZ5KKaWeDNpsGw8YY7I/7m1QSikV\nc09B7Ixfo22VUkqphCDe1zyVUkolHMLTUSvT4KmUUspzBLd3+nrSPA0XCEoppZRHac1TKaWURz35\n9U4NnkoppTxIeDp+qqLNtkoppVQsafBUSinlUeKhxwPXIzJdRM6LyO9OaUNF5LQ9h/NeEanj9N67\nIhIoIodEpKZTei07LVBEBsZkHzV4KqWU8igRzzxi4Fuglpv0scaYovZjubVN8jzWZCIF7TxfiYi3\nPRXmRKA28DzQyl42WtrnqZRSKkEyxmwQkewxXLwBMMsYEwIcF5FAoLT9XqAx5hiAiMyyl/0jusK0\n5qmUUsqDBBHPPID0IrLT6dE9hhvRR0T22826ERPSPgv847TMKTstqvRoac1TKaWUx3j4DkMXjTEl\nY5lnEvAxYOy/nwOdPbdJFg2eSimlPOpx3mHIGPOv03ZMBZbaL08DWZ0WzWKnEU16lLTZViml1BND\nRPydXjYCIkbiLgZaikhiEckB5AG2AzuAPCKSQ0QSYQ0qWvyg9WjNUymllEc9qnqniMwEqmD1jZ4C\nhgBVRKQoVrPtCaAHgDHmoIjMxhoIFAq8aowJs8vpA/wKeAPTjTEHH7RuDZ5KKaU85xHeGN4Y08pN\n8tfRLP8p8Kmb9OXA8tisW5ttlVJKqVjSmqdSSimP0fk8lVJKqTjQ+TyVUkop5UJrnkoppTzqya93\navBUSinlYU9Bq6022yqllFKxpTVPpZRSHmONtn3yq54aPJVSSnmUNtsqpZRSyoXWPJVSSnmQINps\nq5RSSsWONtsqpZRSyoXWPJVSSnmMjrZVSimlYkuejmZbDZ4xYIC7oeGPezMSnLBw87g3IUEKXDvm\ncW9CgpWhZZRTOaoohBy7+Lg3IUHS4KmUUsqjtOaplFJKxdLT8FMVHW2rlFJKxZLWPJVSSnmMAF5P\nfsVTg6dSSinP0mZbpZRSSrnQmqdSSimP0tG2SimlVCxps61SSimlXGjNUymllMfoaFullFIq1p6O\n+Ty12VYppZSKJa15KqWU8hydVUUppZSKvacgdmqzrVJKKRVbWvNUSinlMdZo2ye/7qnBUymllEc9\n+aFTm22VUkqpWNOap1JKKc96CqqeGjyVUkp5lN4kQSmllFIuNHgqpZTyKBHPPB68HpkuIudF5Hen\ntLQiskpEjth/09jpIiLjRSRQRPaLSHGnPB3s5Y+ISIeY7KMGT6WUUh4lHnrEwLdArfvSBgKrjTF5\ngNX2a4DaQB770R2YBFawBYYAZYDSwJCIgBsdDZ5KKaUSJGPMBuDyfckNgO/s598BDZ3SvzeWACC1\niPgDNYFVxpjLxpgrwCpcA7ILHTCklFLKszw3Xii9iOx0ej3FGDPlAXkyGWPO2s/PAZns588C/zgt\nd8pOiyo9Who8lVJKeYzV5Oqx6HnRGFMyrpmNMUZEjKc2xpk22yqllHqS/Gs3x2L/PW+nnwayOi2X\nxU6LKj1aGjyVUkp5jodG2v6H2+MuBiJGzHYAFjmlt7dH3ZYFrtnNu78CNUQkjT1QqIadFi1ttlVK\nKeVRj+oWCSIyE6iC1Td6CmvU7Ahgtoh0AU4Cze3FlwN1gEAgCOgEYIy5LCIfAzvs5T4yxtw/CMmF\nBk+llFIJkjGmVRRvVXOzrAFejaKc6cD02Kxbg6dSSinPevLvzqd9ng9D7x5dyJktM2VKFHF5b/JX\nX1LihecpXbwwg997B4CTJ0+QMU0yKpQpToUyxXmjb68oy27XqhnHjx8DoFH92pQvXYzSxQvzRt9e\nhIWFuSz/xZjRjnLLlChC6mS+XL58mYsXLlCjamXKlCjC0sULHcu3bNaQs2fOOF6/P3AA69etifOx\niKlTp/6hXu1qlC1RmHIlizB54njHe4Pfe5vSxQpSoXQx2rZswrWrVwG4e/cuvbp1onypopQpXogx\no0a4LdsYQ/3a1bl+/ToARQrkonypolQqW4KXKpZxm+fwob+o8VIFMqVJyoRxnzvSL164QK3qlSlX\n8gWWLVnkSG/dvBFnz947boPfHcCGR3DcIoSFhVGzcmk6tGjoSHuzbw9erliS6hVK0L1DS27dvBkp\nz7LFC8iSJjH79uxyW+a/5846yrtz5w79X+1GtfLFebliSbZsWh/ltkyfMpEXSxemarmifPLBuwDs\nCNhC9QolqPNSOY4dPQLAtWtXad24DuHh4Y68LRvW4urVK3E7CG4k9vVm48gGbBvTmF1fNGVQy+KR\n3h/apiT7JzZnz4Sm9H6lIAD9GhYhYExjAsY0ZucXTbg5twtpkid+YFnORnUuS4XnMwNQpfAzbBnd\niIAxjVk9rB45M6cEoMLzmdkyuhE35nahUbkcUZbl6+PFl70qsn9ic/ZOaEbDstkB6FWnIDu/aMKC\nQTXx9bG+yssXyMTITmUdedOnTMKiwQ/8yaKHicf+xWcaPB+CNu06MH/Rcpf0DevXsnzpYrZs38P2\n3Qd47Y03He/lyJmLzdt2s3nbbsZNmOS23D//OEhYWBg5cuQE4Lsff2bL9j1s27WfixcusGDeHJc8\nr/d/y1Hu0I8+pWKlF0mbNi1zZs+ic7furN0YwFdfWoHql2VLKPJCMfyfecaRv0evPowZ/dl/Oh4x\n4ePtwyfDRhGw6wAr125m2pRJ/PXnHwC8VLU6W3bsY/P2PeTKnYcxo60guXD+XELuhLBlx17WbtrO\nt9On8vfJEy5lr/x1OYUKFyFlypSOtCW//MbGgF2s3bTN7fakSZOWEaPH0ef1/pHS582ZRacuPVi9\nYSuTJn4BwC/Ll1DkhaL4+987bt169WHc5yP/0zGJja8nTyB33vyR0oZ+OopVm3by2+ZdPJslK99M\nvXde3bxxg+mTv6RYydJRljll4he07tAZgBnffQ3A6i27mblgOR8PeidS0IuweeM6Vi5fwsqNO1mz\ndS89+/YD4H8Tx/H97EUMHTaaH7+ZCsD40cPp0/8dvLzufQ01adGG76f9L45HwVXI3TBqfbCMMv3n\nU6b/PGoUy0rpvBkBaFc1L1nSJeeFPrMp1ncuczYdBWDswv2U7T+fsv3n88EPO9j4xzmu3AyJtixn\naVMkpnTejGz+45y1nz0r0mncWsr2n8/PGwIZ2KwYAP9cuEn3Cev5ecPRaPfhnaZFuXDtNkVenU2x\n1+aw8aD1E8aWlXNR6o15BPx1npeLZgFgYLPiDJ+zx5H34vXbnLsSRLn8mdyWreJOg+dDUKFiZdKk\nTeuS/vWUyfR7620SJ04MQIaMrh+86MyeNYNX6tV3vI4IBqGhody5ewd5wPC0ObNn0bR5CwB8fX0I\nDgoiJCQEb29vQkND+erL8bzRf0CkPNmee44rly/z77lzsdrW2Mrs788Lxawr+RQpUpA3X37OnrFG\ni1etXgMfH6uHoVTpspw5baWLCEG3bhEaGsrt4GASJUpEihQpXcqeM2smderWd0mPToaMGSleohS+\nvr6R0n18fQkODuJOSAjeXtZxm/zleF7rd99xy/Yclx/BcQM4c/oUq1f+Quv2nSKlp7DPD2MMt4OD\nI50fo4YNpffrb5I4cZIoy/1lyQKqVKsJwJFDf1K+UhUA0mfISMpUqdzWWH+YPoVX3xjgOMfTZ7DO\ncV9fX24HBxEcHISPjy8njh/lzOlTlK/4YqT8L9euy6J5P8fyCETv1u1Qaxu8vfDx9sLq+oLutQow\nbPZu7JdcuHbbJW/zSrmYvTHwgWU5a1g2Byv3nHK8NsaQ0s86j1ImTcTZy7cA+PvCTX4/eZlwN2U4\n61AtH6Pm7bXLgks3QgDr/Pf19iJpYm/uhoXT6sXcrNz9D1duhkTKv2T7SVpUzh3tOjztMY+2fSQ0\neD5CgYFH2LJ5Ey9VKkftl19i184djvdOnjhOxbIlqP3yS2zZtNFt/oCtWyharESktIb1apErW2aS\nJ09Bw8ZNo1x3UFAQv636lfoNmwDQrEVrli1dTMO6NXnz7YFM/d8kWrZuQ9KkSV3yvlC0GAFbN8dl\nl+Pk75Mn2L9vLyVKuTap/vj9N1SvYTVDNWjUhKTJkpE/VxYK589Bn9f7u71o2RawhRecjpuI0Lh+\nbapUKM2306fGatuaNm/F8qWLaVSvFv0HDOTrKZNo0aptlMdtW8CWWJUfF0Pfe4v3PxyOeLl+nPu/\n2o1i+bIReOQwnbv3BuDAvj2cOX2KajXrRFnm3yePkyp1GkcQLFCoCKtWLCU0NJS/Tx7nwF6rjPsd\nCzzCtq2bqVu9Ik1eqc7e3dbNYfr0e5vXe3Xhy7Gj6NitFyM/HsKA94e65E+dOg0hd0K4cvlSXA6F\nW15eQsCYxvz9bTvW7DvNjiMXAMiROSVNK+Zk06iGLBxci1z+kS+8/BJ583KxLCzceuKBZTkrVyAT\ne45edLzuPXEjCwbXInBqK1pXycPo+ftivO2pkiYCYEjrkmwZ3YifBlQjYyo/ACYtP8j6zxqQNUNy\ntv75L+2r5WPyLwddytgdeMHRhPwoeOq+tvE8dsbP4CkiY0XkDafXv4rINKfXn4tIf/e5Xcr6VkRc\nooqIVBGRpZ7Z4pgJDQ3lyuXLrNmwhY+HfUbHti0xxpA5sz8HD59gU8Auhn02mi4d2zr655ydO3eW\n9OkzREpbuGQFh4+f5k5ISLR9k78sW0LZcuVJaweXVKlSMXfBUtZv3s4LRYuzYvkSGjRqSt/e3WnX\nqhnbArY68mbIkJGzZ89GVbRH3bx5k/atmzN85JhIzawAo0cOw8fHh+YtWwOwa+d2vL28+TPwH/Ye\nDGTi+LGcsPuDnV29cpkUKVI4Xv/y23rWb9nBnAVLmfa/SWzetCHG25cqVSpmz1/C2k3b7OO2lPqN\nmvD6qz3o0KY527fdO27pM2TgnFM/6MPw24plpE+fgSJF3fe/jZk4lV1/niBP3nwsXjCH8PBwPnz/\nbT74JPqm+PPnzpE2fXrH65ZtO+L/zLPUeakcQ999ixKly+Lt7fr1ERYaytUrl1myaiODPhpOr06t\nMcZQsPALLFm1kTlLVvL3ieNkzJwZjKFX5zb07d6RC+f/dZSRPn1GznnwfAsPN5TtP5/cXWdQMk8G\nns9m3fM7sY+JSGrJAAAWdklEQVQ3IXfCqDhgId+s+ov/9YlcC36l1HNs/evfSDW5qMpyljlNUi5e\nv1eL7Vu/EI0+XkHubjP5Yc1hPnPqk3wQH28hS/rkBPz1L+XfWsC2Q+cZ3tG6qJy5PpByby6g87h1\n9K1fiK+W/k7N4lmZMaAaIzuVddTczl8Lxj+t68XdQ/UURM94GTyBzUB5ABHxAtIDBZ3eLw888JJe\nRLwfytbF0TPPPkv9ho0QEUqWKo14eXHp4kUSJ05MunTpAChWvAQ5cuYi8Mhhl/x+fn7cDnFtWkqS\nJAl16tVn2ZLFUa573pyfadqspdv3Rg7/hLfeeY+5s2dStnwFJk/7luGffuh4//bt2/j5Rd285yl3\n796lQ+tmNGvRinoNGkV6b8YP37Hyl2VMmf6Do/lx7uxZVHu5Jr6+vmTImJEyZcuzZ7drU6K3j0+k\n/rlnnrFuW5khY0bq1m/AbqcWgNgYNeIT3nz7XebNmUXZchX4aso3fPbpR473Q27fJomfX5zKjqkd\n27aycsUyyhbJy6td2rF54zr6du8YaRlvb2/qN27O8sULuHnjBof+PEizujUoWyQve3Zuo3PrJi5N\nsEn8/Ai5fS9o+Pj4MHTYaFZu3MH0GfO4fu0aOXPlddmezM8+S+16DRERipUohZeXF5cv3auFGWMY\n//lwXh/wHmM++5T3hw6jdYfOTP/fRMcyISG3SfIQzrdrQXdY//sZahSz+gdPX7rFwoATACwKOEGh\n5yK3WjSrmIs5G933R95flrPgO6Ek9rW+etKnTELh7OkcNdS5m45SNhb9j5duhHDr9l0WBhwHYP7m\nYxTNmT7SMv5pklIyT0aWbD/J6w2K0PbzNVy9dYeXiljneRJfH26HhMZ4nSpm4mvw3AKUs58XBH4H\nbth3gEgMFABSicgeETlgz+mWGEBETojIZyKyG2jmXKiI1BKRv+z3Gj+63bHUrdeADevXAXDkyGHu\n3rlDuvTpuXjhgmOk7PHjxzgaeITs9qAgZ3nz5efYUav/5ebNm46r89DQUFb+spy8+fK75AG4du0a\nmzZt4JV6DVzeCww8wpnTp6hUuQpBQcF4eXkhItwOvu20zGEKPF/oP+37gxhj6NurG3nzFeDV1/pF\neu+3lSsYP240M2YvjNQ8miVLVjauXwvArVu32LljG3ny5nMpO0+evI4a6a1bt7hx44bj+ZrVqyjw\nfEGXPA9yNPAIZ86cpmLlKgQHBSH2cQu+HexYJjDwSJzKjo13h3zCzoPHCNh/mIlf/0CFSlWYMOVb\njDEcP2adK8YYVq1YSu68+UiZKhUHjp4hYP9hAvYfpljJMkyfMS9SszZAzlx5OPX3Scfr4KAggm5Z\nfXUb1v6Gj48PefMXcNmeWnXqs2WjNRL3WOBh7ty5S9p0977s5876kaov1yJNmrQEBwfh5eWFl3gR\nHBzk2Nbz5/8la7bsHjk+6VMmcTR9JknkTbUXsnDo9DUAlmw/wYuF/QGoVNCfwDPXHPlSJvWlYsHM\nLNl+MkZlOTt06qqjCfjKzRBSJk1E7mdSAVD1hSwcOnU1VvuwfMffVC5kDUarUuQZ/joVeTTyB61L\n8vFMq3ncL5E3xhjCjSFpYmucQJ5nUnHwb8+NYI6Jp2G0bbz8nacx5oyIhIpINqxa5lasu9yXA64B\nR4BpQDVjzGER+R7oBYyzi7hkjCkOVsC0/yYBpgJVse4wEe2oBBHpjjXnG1mzZovV9ndq35pNG9dz\n6eJF8ufKxnuDh9C+YxfadehM7x5dKFOiCIkSJWLytG8QETZv2sCnHw/F19cXLy8vxk34ytG86qxm\n7Tps3LCel6pWJ+jWLVo0bcidOyGEh4dTqXIVunTrAcDXUycD0KVbTwCWLl5A1WovkyxZMpcyPx4y\niMEffgJAs+YtadW8MWNHj+T9wUMBqzZ47OhRipeI872ZYyRg62Z+nvkjzxcsTKWy1hf54KEfU6NW\nHd5+83VCQkJoVM/q6yxZugxjx39F1x696dOzC+VKFsEYQ+u2HShU2PXnQTVq1WHTxvXkzJWbC+f/\npW1LqxU/LCyUJs1bOvpQp9ujPDt37cG/585RtVIZbty4jnh5MXnieLbuOuBoSv7kw8EMGvIxAE2a\ntaRNy8Z8MWYk7w4aCljH7fixoxQr/nCPW1SMMfTr1ZUbN66DMRQoVIThn0+Icf6kyZLxXI4cHD8W\nSI6cubl48TxtmtTFy8uLzP7P8MXke78nf+u1nrTr1I0XipWgRduOvNmnO9XKFcM3USLGTZrmaCkI\nDgpi9owfmDF/GQDde79Ou+YNSJQoEV9OtWaQ2r93N8VLlnYMEPuvMqdJytTXXsTbS/DyEuZtPsYv\nO/8GYPS8fXzT7yX61ivMrdt36fXVveb7+mWys3rvaYKcamzRleVsxc5/6FIzP9/+doiwcMOrX21k\n5tvVCQ83XL0VQo8vrfWUyJ2en995mdTJE1OnVDYGtSxBidfnAhAwpjFl+88HYNAP2/n69SqM6lyW\ni9dv02PCvZ8JvZDDarHae8zqI/55w1F2jmvKqYs3GbPA6lt9sbA/K3Y5Txry8MX3wT6eIO5Gi8UH\nIvITsARrAtMxWMGzPFbwLA+kM8ZUtpetBrxqjGksIieAF40xJ+33vgWWYgXM8U556gPdjTF1H7Qt\nxUuUNOs3b/fsDsZBcHAwr9Ssxqq1G/H2fjQt0ksWLWDv3j0MHvLRgxe+T1h4/Di3zp09S69uHVmw\n9IG3q/SYpYsXsm/vbt7/IPbH7dYd19/rPg6/LF3Egb27eXvQhw9e2EM+GNifGrXrUvHFqnHKn7vT\ndw9e6BFYPawejT/5lWtBdx73prDqk7o0G76Sq7fcb0vIuk8Jv3LCY+GuYJHiZtbymI8jiE6RrCl2\n/ZdZVR6m+NpsC/f6PQtjNdsGYNU8ywPrHpD31kPdssfEz8+P9wYPcfxU41EIDQ2l7+sxGpsVb2X2\n96d9p65uB2E9LKGhofR5LWEft9p1G5Al23OPdJ35ChSMc+CMTwZ+E0DWDMkf92aQPmUSxi8+EGXg\nfFiegvFC8bPZ1rYFeAs4ZowJAy6LSGqsPtC+wJsiktsYEwi0A6K+5YnlLyC7iOQyxhwForonYrxW\n/eWaj3R9jZo0e/BCCcCj3o/ofjaUkLRu3/mRrq9Nhy6PdH0Pi7ufsDwOF6/fjtRv+0gkhMjnAfG5\n5nkAa5RtwH1p14wxp7DuiD9HRA4A4cDk6AozxtzG6sNcZg8YOh/d8koppVRU4m3N065tprwvraPT\n89VAMTf5skeTZwXgfkiqUkopj4jvI2U9Id4GT6WUUgmP8HSMto3PzbZKKaVUvKQ1T6WUUh71FFQ8\nNXgqpZTysKcgemqzrVJKKRVLWvNUSinlUTraVimllIolHW2rlFJKKRda81RKKeVRT0HFU4OnUkop\nD3sKoqc22yqllFKxpDVPpZRSHmNNqvLkVz01eCqllPIc0dG2SimllHJDa55KKaU86imoeGrwVEop\n5WFPQfTUZlullFIqlrTmqZRSyoNER9sqpZRSsaWjbZVSSinlQmueSimlPEZ4KsYLafBUSinlYU9B\n9NRmW6WUUiqWNHgqpZTyKPHQvweuR+SEiBwQkb0istNOSysiq0TkiP03jZ0uIjJeRAJFZL+IFP8v\n+6jBUymllEeJeOYRQy8ZY4oaY0rarwcCq40xeYDV9muA2kAe+9EdmPRf9lGDp1JKqSdJA+A7+/l3\nQEOn9O+NJQBILSL+cV2JBk+llFIeJR56AOlFZKfTo/t9qzLAShHZ5fReJmPMWfv5OSCT/fxZ4B+n\nvKfstDjR0bZKKaU8x7NTkl10ao51p6Ix5rSIZARWichfzm8aY4yIGI9tjROteSqllEqQjDGn7b/n\ngQVAaeDfiOZY++95e/HTQFan7FnstDjR4KmUUsrDPNhwG9UaRJKJSIqI50AN4HdgMdDBXqwDsMh+\nvhhob4+6LQtcc2rejTVttlVKKeUxwiO7t20mYIFYK/MBZhhjVojIDmC2iHQBTgLN7eWXA3WAQCAI\n6PRfVq7BUymlVIJjjDkGvOAm/RJQzU26AV711Po1eCqllPKop+DufBo8Y2LP7l0XU/p5n3zc2xGF\n9MDFx70RCZAet7jR4xY38fm4PefpAp+GKck0eMaAMSbD496GqIjIzgcM5VZu6HGLGz1ucaPH7cmj\nwVMppZRHxeS+tAmdBk+llFKe9eTHTv2d5xNgyuPegARKj1vc6HGLGz1uTxiteSZwxhj9UMaBHre4\n0eMWN0/bcXsKKp4aPJVSSnlOLKcTS7C02VYppZSKJQ2ej4iIhNmzne8Tkd0iUj6O5fQUkfae3r6E\nQETeF5GD9izwe0WkTBTLlRSR8U6vfUXkuJ1nr4icE5HTTq8TxXD9VURkqaf253GK6bGMZZlV4npe\nJxQiMlZE3nB6/auITHN6/bmI9I9hWd+KSFM36Qn+PBMP/YvPtNn20Qk2xhQFEJGawHDgxdgWYoyZ\n7OkNSwhEpBxQFyhujAkRkfSA26BnjNkJ7HRKqggsNcb0tcsaCtw0xox+uFsdP8XmWMaiTB+gCnAT\n2PKfNzL+2ox1r9RxIuKFdfODlE7vlwf6PagQEfF+OJsXT8TvuOcRWvN8PFICV8D1KlNEvhSRjvbz\nESLyh107GG2nDRWRt+zn60TkMxHZLiKHRaSSne4tIqNEZIedt4ed7i8iG+yaxu8iUsle9lv79QER\neeAH/zHxx5rbLwTAGHPRGHNGREqJyBa7Rr9dRFK4uXKvBfzirlARKSEi6+3JdH91msoot4j85tRS\nkMvOklxE5orIXyLyk0iC7N2J6lieEJGR9nmwXURyA4hIdhFZY59Lq0Ukm53+rYhMFpFtwGygJ9DP\nPr8qiUgz+7zaJyIbHtfOetgWoJz9vCDWLB43RCSNiCQGCgCpRGSPfRyn2+nYx/czEdkNNHMuVERq\n2efUbqDxo9sdFVda83x0/ERkL5AE68uranQLi0g6oBGQ357QNXUUi/oYY0qLSB1gCFAd6II13U4p\n+4O7WURWYn0ofzXGfGpf+SYFigLPGmMK2euNaj2P20rgAxE5DPwG/Axstf+2MMbsEJGUQLCbvC8B\nH96fKCK+wASggTHmgoi0AD4FOgM/ASOMMQtEJAnWhWZWoBjWl+YZrFpIBWCTR/f04XM5lsaY9fZ7\n14wxhcXqGhiHVUOdAHxnjPlORDoD44GG9vJZgPLGmLD7a/QicgCoaU9WHF/Pq1ixLzJC7QuI8ljn\n4LNYAfUacASYBlQzxhwWke+BXljHEuCSMaY4WAHT/psEmIr1nRCIdU4naAnxijK2tOb56AQbY4oa\nY/Jj1YS+f0Ct5RpwG/haRBpjTaHjznz77y4gu/28Bta8dXuBbUA6IA+wA+hkf8kVNsbcAI4BOUVk\ngv1hvh7XHXyYjDE3gRJAd+AC1hdMD+CsMWaHvcx1Y0yocz4ReRa4bIxxd/zyAYWwZqDfCwwCsog1\nR+CzxpgFdrm3nfJvN8acMsaEA3u5d8wTDHfHMqK1A5jp9DeihlUOmGE//wGrGTzCHGNMWBSr2gx8\nKyLdgCepmXILVuCMCJ5bnV6fAo4bYw7by34HVHbK6y4w5rfzHLFn/vjxYW34oxIx4va/PuIzrXk+\nBsaYrXY/UwYglMgXMUnsZUJFpDTW1DpNgT64r62G2H/DuPf/KUBfY8yv9y8sIpWBV7C+1MYYY74X\nkReAmljNbs2xal7xjv0lvQ5YZ9dqYjK9UC3A5TjYBDhojCkXKdGeYDcKIU7PnY95guLmWEZMHmyc\nF4tBUbeiWUdPsQYivQLsEpES9nRRCd1mrEBZGKvZ9h/gTawLz3VAk2jyRnm8nhzxf7CPJ2jN8zEQ\nkfxYV+KXsCZrfV5EEttNW9XsZZIDqYwxy7EGILjMWxeNX4FedrMkIpJXrFnXnwP+NcZMxWpaKm4H\ncS9jzDysmldxz+ylZ4lIPhHJ45RUFPgT8BeRUvYyKcQauOIsyv5O4BCQQawBNBGjcgvaNfJTItLQ\nTk8sIkk9uT+PUxTHMmLWoBZOf7faz7cALe3nbYCNURR9A3BceIhILmPMNmPMB1g13Kwe2Pz4YAtW\nc/ZlY0yYMeYykBqrhj4PyB7RXwy0A9a7L8bhLztPRL96q4ewzcrDEuRVcwIV0ecJVo2ng331/4+I\nzMa6gj0O7LGXSQEssvtDBIjR8HfbNKzmxN120/AFrD6qKsAAEbmLNSqyPVZ/zTdijRwEeDduu/fQ\nJQcm2BcYoVh9Q92Bb+x0P6z+zuoRGex+3dzGmL/cFWiMuSPWTwXGi0gqrM/DOOAg1pfe/0TkI+Au\n9w3wSOCiOpZ1gTQish+rhh3xJd4X6xwZgHUudYqi3CXAXBFpYOfpZwdpAVYD+x7S/jxqB7BG2c64\nLy25MeaUiHQC5tgXcjuAaEfIG2Nui0h3YJmIBGFdnETX+hGvCfG/ydUTxGpiV+rJIyIVgbbGmJ6P\ne1sSAhE5AZQ0xsTXeSdVAlCseEmzZtM2j5SVNpnPrvg6lZvWPNUTyxiziYQ3ElYplQBo8FRKAWCM\nyf64t0E9GZ6GZlsNnkoppTxKR9sqpZRSyoXWPJVSSnlOArjBgSdozVM99eTejDe/i8ic//KbTnGa\nKUNEponI89EsG6dZSOx7pKaPafp9y9yM5boc91JWKibEg4/4TIOnUvdunVgIuIN1pyUHNzdeiBFj\nTFdjzB/RLFIF6041SqkERoOnUpFtBHLbtcKNIrIY+EOinqlGxJoJ55CI/AZkjChIrFlvStrPa4k1\nO8s+sWYmyY7rLCQZRGSevY4dIlLBzptORFaKNf/mNGJwUS4iC8WaKeag/QN85/fG2umrRSSDnZZL\nRFbYeTbad8FSKm6egqqn9nkqZbNrmLWBFXZScaCQMea4HYDczVRTDOsG888DmYA/gOn3lZsBa9aM\nynZZaY0xl0VkMpFnIZkBjDXGbBJr1o5fsaa4GgJsMsZ8JCKvYM2a8yCd7XX4ATtEZJ59X9lkwE5j\nTD8R+cAuuw8wBehpjDki1v1ov+IBM/8oFZWnYbStBk+lIt86cSPwNVZz6nZjzHE7vQZQJKI/E0iF\nNVNNZWCmfavFMyKyxk35ZYENEWXZ90J1pzrWfY4jXqcU6x7HlbHneDTGLBORKzHYp9dEpJH9PKu9\nrZeAcO7N7PEjMN9eR3msW8pF5E8cg3Uo9dTS4KmU3efpnGAHEecZMNzOVCPWPKqe4gWUNcbcdrMt\nMSYiVbACcTljTJCIrMOerccNY6/36v3HQKm40tG2SqkIbmeqATYALew+UX+sibfvFwBUFpEcdt60\ndnqkWUiwJqnuG/FCRCKC2QagtZ1WG0jzgG1NBVyxA2d+rJpvBC+sKe6wy9xkjLkOHBeRZvY6RKxp\n6pSKk6egy1ODp1IxNA2rP3O3iPwO/A+r5WYBcMR+73vuTePlYIy5gDVryXwR2ce9ZtMlQKOIAUPA\na0BJe0DSH9wb9fshVvA9iNV8+/cDtnUF4CMifwIjsIJ3hFtAaXsfqgIf2eltgC729h0EGsTgmCj1\n1NJZVZRSSnlM8RIlzaaAHR4pK1kiL51VRSml1NPhaRhtq822SimlVCxpzVMppZTHCE/HaFvt81RK\nKeUxIrICiPYey7Fw0RhTy0NleZQGT6WUUiqWtM9TKaWUiiUNnkoppVQsafBUSimlYkmDp1JKKRVL\nGjyVUkqpWPo/B7gH4pl4aM4AAAAASUVORK5CYII=\n","text/plain":["<Figure size 504x504 with 2 Axes>"]},"metadata":{"tags":[]}},{"output_type":"stream","text":["              precision    recall  f1-score   support\n","\n","           0       0.84      0.69      0.76      4500\n","           1       0.79      0.79      0.79      4500\n","           2       0.78      0.93      0.85      4500\n","           3       0.82      0.82      0.82      4500\n","\n","    accuracy                           0.80     18000\n","   macro avg       0.81      0.80      0.80     18000\n","weighted avg       0.81      0.80      0.80     18000\n","\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"yeiD1T_QZpdk","colab_type":"text"},"source":["## Inference"]},{"cell_type":"code","metadata":{"id":"z7G7vuSTZHkQ","colab_type":"code","colab":{}},"source":["import collections"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"362Bl2chXDOA","colab_type":"code","colab":{}},"source":["def get_probability_distribution(y_prob, classes):\n","    results = {}\n","    for i, class_ in enumerate(classes):\n","        results[class_] = np.float64(y_prob[i])\n","    sorted_results = {k: v for k, v in sorted(\n","        results.items(), key=lambda item: item[1], reverse=True)}\n","    return sorted_results"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"CLP2Vzp3Zwth","colab_type":"code","outputId":"a8b26809-bc68-4bf8-86cd-aae4d3adf5a4","executionInfo":{"status":"ok","timestamp":1584551606237,"user_tz":420,"elapsed":505,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":85}},"source":["# Inputs\n","texts = [\"The Wimbledon tennis tournament starts next week.\"]\n","num_samples = len(texts)\n","X_infer = np.array(X_tokenizer.texts_to_sequences(texts))\n","print (f\"{texts[0]} \\n\\t→ {decode(X_infer[0], X_tokenizer)} \\n\\t→ {X_infer[0]}\")\n","print (f\"len(X_infer[0]): {len(X_infer[0])} words\")\n","y_filler = np.array([0]*num_samples)"],"execution_count":126,"outputs":[{"output_type":"stream","text":["The Wimbledon tennis tournament starts next week. \n","\t→ the wimbledon tennis tournament starts next week \n","\t→ [   10 13592   869  3141   782   207   223]\n","len(X_infer[0]): 7 words\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"q1gFlI5MZ143","colab_type":"code","colab":{}},"source":["# Dataset\n","infer_set = TextDataset(X=X_infer, y=y_filler, batch_size=BATCH_SIZE, max_filter_size=max(FILTER_SIZES))"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"UFE4sp_7aHTq","colab_type":"code","colab":{}},"source":["# Iterate over val batches\n","probabilities = []\n","for i, batch in enumerate(infer_set.generate_batches()):\n","    \n","    # Cast\n","    X, y = batch\n","    X = X.to(device)\n","    y = y.to(device)\n","\n","    # Forward pass\n","    with torch.no_grad():\n","        y_pred = model(X, apply_softmax=True)\n","\n","    # Save probabilities\n","    probabilities.extend(y_pred.cpu().numpy())"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"bGi_NvbBaMap","colab_type":"code","outputId":"bc853c9d-c477-4c77-d197-d613a9617645","executionInfo":{"status":"ok","timestamp":1584551609255,"user_tz":420,"elapsed":751,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":221}},"source":["# Results\n","results = []\n","for index in range(num_samples):\n","    results.append({\n","        'raw_input': texts[index],\n","        'preprocessed_input': decode(indices=X_infer[index], tokenizer=X_tokenizer),\n","        'probabilities': get_probability_distribution(probabilities[index], y_tokenizer.classes_)\n","                   })\n","print (json.dumps(results, indent=4))"],"execution_count":129,"outputs":[{"output_type":"stream","text":["[\n","    {\n","        \"raw_input\": \"The Wimbledon tennis tournament starts next week.\",\n","        \"preprocessed_input\": \"the wimbledon tennis tournament starts next week\",\n","        \"probabilities\": {\n","            \"Sports\": 1.0,\n","            \"World\": 2.527271469356107e-13,\n","            \"Business\": 2.4281697622569975e-16,\n","            \"Sci/Tech\": 3.0714927373986137e-19\n","        }\n","    }\n","]\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"Y4-WkjN595lO","colab_type":"text"},"source":["# Interpretability"]},{"cell_type":"markdown","metadata":{"id":"Uo0FqqEY98El","colab_type":"text"},"source":["Recall that each our unique filter sizes (2, 3 and 4) act as n-gram feature detectors. When these filters convolve on our embedded input (`N`, `max_seq_len`, `embedding_dim`), they produce feature maps which are shape ((`N`, `max_seq_len`, `num_filters`) for each filter size. Since we used `SAME` padding with stride=1, our feature maps have the same length as our inputs ('max_seq_len') which you can think of as what the filters extracted from each n-gram window. When we apply 1d global max-pooling we're effectively extracting the most relevant information from the feature maps. We can inspect the trained model at the pooling step to determine which n-grams were most relevant towards the prediction."]},{"cell_type":"code","metadata":{"id":"Zv2uqi6mOe9Z","colab_type":"code","colab":{}},"source":["import seaborn as sns\n","from statistics import mode"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"M-aGz2BgCCKq","colab_type":"text"},"source":["We're going to copy the same model structure as before but now we'll stop just after convolution since those are the outputs we care about."]},{"cell_type":"code","metadata":{"id":"_nzdZ2_tBsfc","colab_type":"code","colab":{}},"source":["class ConvOutputsModels(nn.Module):\n","    def __init__(self, embedding_dim, vocab_size, num_filters, \n","                 filter_sizes, hidden_dim, dropout_p, num_classes,\n","                 pretrained_embeddings=None, freeze_embeddings=False, \n","                 padding_idx=0):\n","        super(ConvOutputsModels, self).__init__()\n","\n","        # Filter sizes\n","        self.filter_sizes = filter_sizes\n","        \n","        # Initialize embeddings\n","        if pretrained_embeddings is None:\n","            self.embeddings = nn.Embedding(embedding_dim=embedding_dim,\n","                                          num_embeddings=vocab_size,\n","                                          padding_idx=padding_idx)\n","        else:\n","            pretrained_embeddings = torch.from_numpy(pretrained_embeddings).float()\n","            self.embeddings = nn.Embedding(embedding_dim=embedding_dim,\n","                                           num_embeddings=vocab_size,\n","                                           padding_idx=padding_idx,\n","                                           _weight=pretrained_embeddings)\n","        \n","        # Freeze embeddings or not\n","        if freeze_embeddings:\n","            self.embeddings.weight.requires_grad = False\n","        \n","        # Conv weights\n","        self.conv = nn.ModuleList(\n","            [nn.Conv1d(in_channels=embedding_dim, \n","                       out_channels=num_filters, \n","                       kernel_size=f) for f in filter_sizes])\n","        \n","        # FC weights\n","        self.dropout = nn.Dropout(dropout_p)\n","        self.fc1 = nn.Linear(num_filters*len(filter_sizes), hidden_dim)\n","        self.fc2 = nn.Linear(hidden_dim, num_classes)\n","\n","    def forward(self, x_in, channel_first=False, apply_softmax=False):\n","        \n","        # Embed\n","        x_in = self.embeddings(x_in)\n","\n","        # Rearrange input so num_channels is in dim 1 (N, C, L)\n","        if not channel_first:\n","            x_in = x_in.transpose(1, 2)\n","            \n","        # Conv outputs\n","        z = []\n","        max_seq_len = x_in.shape[2]\n","        for i, f in enumerate(self.filter_sizes):\n","            # `SAME` padding\n","            padding_left = int((self.conv[i].stride[0]*(max_seq_len-1) - max_seq_len + self.filter_sizes[i])/2)\n","            padding_right = int(math.ceil((self.conv[i].stride[0]*(max_seq_len-1) - max_seq_len + self.filter_sizes[i])/2))\n","            \n","            # Conv\n","            _z = self.conv[i](F.pad(x_in, (padding_left, padding_right)))\n","            z.append(_z.cpu().numpy())\n","        \n","        return z"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"XfWHwZ7DB2gf","colab_type":"code","outputId":"5363a5c9-4e7c-484b-db70-03e965212a09","executionInfo":{"status":"ok","timestamp":1584551616473,"user_tz":420,"elapsed":1059,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":204}},"source":["# Initialize model\n","model = ConvOutputsModels(embedding_dim=EMBEDDING_DIM,\n","                          vocab_size=vocab_size,\n","                          num_filters=NUM_FILTERS,\n","                          filter_sizes=FILTER_SIZES, \n","                          hidden_dim=HIDDEN_DIM, \n","                          dropout_p=DROPOUT_P, \n","                          num_classes=len(classes),\n","                          pretrained_embeddings=embedding_matrix,\n","                          freeze_embeddings=FREEZE_EMBEDDINGS).to(device)\n","print (model.named_parameters)"],"execution_count":132,"outputs":[{"output_type":"stream","text":["<bound method Module.named_parameters of ConvOutputsModels(\n","  (embeddings): Embedding(29795, 100, padding_idx=0)\n","  (conv): ModuleList(\n","    (0): Conv1d(100, 50, kernel_size=(2,), stride=(1,))\n","    (1): Conv1d(100, 50, kernel_size=(3,), stride=(1,))\n","    (2): Conv1d(100, 50, kernel_size=(4,), stride=(1,))\n","  )\n","  (dropout): Dropout(p=0.1, inplace=False)\n","  (fc1): Linear(in_features=150, out_features=100, bias=True)\n","  (fc2): Linear(in_features=100, out_features=4, bias=True)\n",")>\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"4fL_exZ2CMP0","colab_type":"text"},"source":["Since we already trained our model, we'll transfer those weights to our new model."]},{"cell_type":"code","metadata":{"id":"Q24ZsZofCkNV","colab_type":"code","outputId":"1ea15888-c1d8-4c09-fc2e-7edeccf9233b","executionInfo":{"status":"ok","timestamp":1584551618116,"user_tz":420,"elapsed":269,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":34}},"source":["# Transfer weights\n","model.load_state_dict(glove_finetuned_model.state_dict())"],"execution_count":133,"outputs":[{"output_type":"execute_result","data":{"text/plain":["<All keys matched successfully>"]},"metadata":{"tags":[]},"execution_count":133}]},{"cell_type":"code","metadata":{"id":"pZQY75xXC4rZ","colab_type":"code","colab":{}},"source":["# Iterate over val batches\n","conv_outputs = []\n","for i, batch in enumerate(infer_set.generate_batches()):\n","    \n","    # Cast\n","    X, y = batch\n","    X = X.to(device)\n","    y = y.to(device)\n","\n","    # Forward pass\n","    with torch.no_grad():\n","        z = model(X, apply_softmax=True)\n","    \n","    # Save\n","    conv_outputs.extend(z)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"RyC7FJndIFaE","colab_type":"code","outputId":"8f8c47da-e636-4d79-eaf3-0dcc46b31eb4","executionInfo":{"status":"ok","timestamp":1584551629600,"user_tz":420,"elapsed":822,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":494}},"source":["# Visualize bi-gram filters\n","tokens = decode(X_infer[0], X_tokenizer).split(' ')\n","sns.heatmap(conv_outputs[1][0], xticklabels=tokens)"],"execution_count":136,"outputs":[{"output_type":"execute_result","data":{"text/plain":["<matplotlib.axes._subplots.AxesSubplot at 0x7fd1613de6a0>"]},"metadata":{"tags":[]},"execution_count":136},{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAZQAAAHMCAYAAADyGF46AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjAsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8GearUAAAgAElEQVR4nO3dd5hlVZ3u8e/buZvuJmeQJnsVMNBg\nGgNBBQOYGNRRFEMrdxCdmWsa5jHMjAo6BkavSiuCihcjWUSSgIksOUkUEOkBBBq6obu6fvePvQtO\nt121z+7a56y9Tr0fn/Nwap+za79dVtWqvcJvKSIwMzMbr0mpA5iZ2WBwg2JmZo1wg2JmZo1wg2Jm\nZo1wg2JmZo1wg2JmZo2Y0usLnL/xAVnNSz5jZs+/JI17xvLJqSPUdsnUZakj1HLB0jtTR6ht35nb\npI5Q2y4Zfi8fdM/xaupzLb//tsZ+X07dYJvGcnWr8renpKcD+wObl4fuAU6NiBt6GczMzPIyZpeX\npI8CPwQEXFI+BJwg6WO9j2dmNoEMr2jukUDVHcq7gWdGxPLOg5K+BFwHHLG6kyQtABYA/POc5/La\nDG+9zcz6LoZTJxiXqkH5YWCz1RzftHxttSJiYUTMj4j5bkzMzLo0PNzcI4GqO5QPAedK+iNwV3ns\nacB2wKG9DGZmZnkZs0GJiDMl7QDszsqD8pdGRJpOOjOzARWZd3lVzvKK4l940ZpeYIO1lq7pqUn8\n2155TWcFmPrCZ6eOUNsLP3916gi17PnEdqkj1HZy5PWzB7DNpFmpI6SVqKuqKV7YaGZmjchvFZ+Z\n2aAa9C4vMzPrk0TrR5riLi8zM2uE71DMzNrCXV5mZtaIzGd59bxB+fSKvKqHfuuR5dVvapkV192U\nOkJtW37ieakj1LL2wvNTR6htt4enpo5Q2zEPzkwdwcbBdyhmZi0x8AsbzcysTzLv8qqc5SXp6ZL2\nkjR7leP79C6WmZnlpmo/lMOAU4APANdK2r/j5c/2MpiZ2YQTw809Eqjq8novsGtEPCppHvBTSfMi\n4iiKjbZWq3M/lOeutwvbzJ7XTFozs0E24AsbJ0XEowARcQfwMmDfcoOtURuUzv1Q3JiYmU0MVQ3K\nfZKeLGVbNi6vATYAdu5lMDOzCWfAu7wOAoY6D0TEEHCQpKO7ucCuzFnDaGn85Q/5rUM59aK8vsYA\nS35yS+oItdzAuqkj1JZjXaXrnrgtdYTa/r3JT5b5LK+qDbbuHuO13zYfx8zMcuV1KGZmbeGFjWZm\n1ojMu7xy7GY1M7MW8h2KmVlLROS9DsUNiplZW3gMZWw7P55Xi7vhTo+ljlDbB97x/NQRahu+9vrU\nEWoZunNx6gi13far2dVvapnzpm2bOoKNg+9QzMzaIvNBeTcoZmZtkXmXV+1ZXpK+14sgZmaWtzHv\nUCSduuohYA9J6wBExH69CmZmNuFkXm24qstrC+B64NtAUDQo84EvjnVSZ/n6Q+fMZ5+Z240/qZnZ\noBvwLq/5wOXA4cDDEXE+sDQiLoiIC0Y7qbN8vRsTM7OJoao45DDwZUk/Kf97X9U5Zma2hibCLK+y\n6vABkl4NPFLnAndNm7wmuZLRZZumjlDbCddemTpCbZ/d6OHUEWo5ZtEmqSPUtuNQ9XvaZouI1BHS\nyrzLq9bdRkT8HPh5j7KYmVnG3H1lZtYWE6HLy8zM+iDzBsXl683MrBG+QzEzawmXrzczs2a4y8vM\nzKwPdyj7bXlPry/RqCcey++m7YhZ+d0m33jbhqkj1LKnlqWOUNupM/L7Xp4dE/xv3D6uQ5G0JfA9\nYGOK0loLI+Ko8XzO/L7jzMwGVX+7vIaAf4mIKyTNAS6XdHZErPHud2P+OSDpeZLmls9nSvq0pNMk\nHSlp7TW9qJmZpRUR90bEFeXzxcANwObj+ZxV95ffAZaUz48C1gaOLI8dO54Lm5nZKmK4sYekBZIu\n63gsGO2ykuYBzwEuHk/8qi6vSRExUhFofkQ8t3z+G0mjFpDqLF9/5DY78LaNNxtPRjOziaHBLq+I\nWAgsrHqfpNnAz4APRUStWo2rqrpDuVbSweXzqyTNLwPsACwf7aTO8vVuTMzM2knSVIrG5AcRceJ4\nP1/VHcp7gKMk/RtwP/B7SXcBd5WvmZlZU/o7y0vAMcANEfGlJj5n1X4oDwPvLAfmty7ff3dE3Nft\nBf5wa15lv0+fmd8U3OUP57cY6lnTpqaOUMu1k59IHaG2jTNcZrb9cqWOkFZ/Z3m9CHg7cE3HEMa/\nRsQZa/oJu90P5RHgqjW9iJmZtUtE/IZiW/fGeB2KmVlbZF56xQ2KmVlbZL5jY36drGZm1kq+QzEz\nawt3eZmZWSMy7/Jyg2Jm1ha+Qxnb02Y92utLNOqwKfmtQ9ly/7zWdAA88rsHUkeo5eX3z0wdobal\nj+dXcn9IHtbNme9QzMzaYpC7vCRNA94M/DkizpH0VuCFFGWOF0bEqPW8zMyspgHv8jq2fM8sSe8A\nZgMnAnsBuwPv6G08MzPLRVWDsnNE7CJpCnAPsFlErJB0PGOUYuksX/+J9XfmTXO2aiywmdnAyvwO\npWoEbFLZ7TUHmEWxwRbAdGDUkeDO8vVuTMzMuhTR3COBqjuUY4AbgcnA4cBPJN0GPB/4YY+zmZlZ\nRqrK139Z0o/K53+W9D1gb+BbEXFJPwKamU0YmXd5VU4bjog/dzx/CPhpnQs8tHT6GsRKZ+aUoeo3\ntcxVx+c3+/t7MzZIHaGWO1bktZ4K4J5lXW9b1BofnLpd6gi17dbkJ8u8QfEqIjMza0R+f9qamQ2q\nQV7YaGZmfeQuLzMzM9+hmJm1R6L1I01xg2Jm1haZd3n1vEGZPS2v+pHHTcqvTPm8Sfn9XbCCx1NH\nqOVVWj91hNpum7l29ZtaZt0n8v4LfaLL7zeRmdmg8h2KmZk1IvNpw57lZWZmjehJgyJpgaTLJF32\ns0fv6MUlzMwGTgxHY48UxmxQJK0t6QhJN0p6UNIDkm4oj60z2nmd5evfOHte46HNzAbS8HBzjwSq\n7lB+DPwVeFlErBcR6wN7lMd+3OtwZmaWj6pB+XkRcWTngYj4C3CkpHf1LpaZ2QSU+aB8VYNyp6SP\nAN+NiPsAJG0MvBO4q5sLPDE0eVwB++3DWy5KHaG2abNXpI5Q22k3bZk6Qi0zMvw5v3Vafms6pmS+\nUnzcEo19NKWqy+tAYH3ggnIM5UHgfGA94IAeZzMzs4xU7dj4V+Cj5WMlkg4Gju1RLjOziSfzhY3j\nmTb86cZSmJlZ9rO8xrxDkXT1aC8BGzcfx8zMclU1KL8x8EqKacKdBPyuJ4nMzCaqzCclVDUopwOz\nI+LKVV+QdH5PEpmZTVSZj6FUDcq/e4zX3trNBY6eprqZkjr3j/+TOkJtk5VfSba7Ft+SOkItW83N\nr4f3TdN2TB2httNnLEsdobbXpQ7QIq42bGbWFpmvQ3GDYmbWFpmvlM+vr8TMzFrJdyhmZm2ReZdX\nVfn6uZI+J+n7kt66ymtfH+O8J/dDuWnx7U1lNTMbaDE83Ngjhaour2Mp1pz8DHizpJ9Jml6+9vzR\nTurcD2XHOVs3FNXMzNqsqstr24h4Y/n8ZEmHA+dJ2q/HuczMJp7Mu7yqGpTpkiZFFFMPIuIzku4B\nLgRmd3OBty3Na5jmA2ttkzpCbXcsXyt1hNq233xx6gi1/G752qkj1DZteeoE9W08lPcv1HEb8Fle\npwF7dh6IiOOAfwHyW4FkZmY9U7VS/iOjHD9T0md7E8nMbILKvMvL5evNzNqij+XrJX1H0iJJ1zYV\n3+XrzcwmpuOArwHfa+oTuny9mVlb9LHLKyIulDSvyc/p8vVmZm3R4CwvSQuABR2HFkbEwsYusBo9\nL19vZmb9VzYePW1AVtXzRSInzez1FZq1aXS1vKZVVkyvfk/bbLB0RuoItayn/Gbf5Lii4bszHk8d\nobZXNfnJMp/lldeqQzOzAZaqBldTXL7ezGwCknQC8HtgR0l3Sxp1iKNbte9QJG0UEYvGe2EzM1tF\nf2d5vaXpz1lVvn69VR7rA5dIWlfSemOc92T5+usW39p0ZjOzwTQczT0SqLpDuR+4c5VjmwNXAAGs\ntpJi5+yCQ+cdmPcok5mZdaWqQfkw8HLgwxFxDYCk2yPCm5yYmTUt82rDVetQvijpR8CXJd0FfJLi\nzqRrm8fUccTrv1uV37TF4/78+9QRavvytLzmk79kvaenjlDbNE1OHaG2F8ac1BHSGvRpwxFxN3BA\nuanW2cCsnqcyM5uAIvMGpetpwxFxKrAHsDeApIN7FcrMzPJTax1KRCyNiJFSxy5fb2bWpEGe5eXy\n9WZmfZT5SnmXrzczs0a4fL2ZWVtkPijv8vVmZm0xyA1KE3Z8Iq8+wU0n51VWHeATu+S3RmLZkrwK\nXd/yQH7bGhw9/bHUEWq7d9KK1BFsHPL6qTYzG2ARvkMxM7MmZN7l5f1QzMysEbUblLKEfdV7nixf\nf9aSW9YsmZnZRJP5wsaq/VCOkLRB+Xy+pNuAiyXdKemlo50XEQsjYn5EzH/FrO0ajmxmNphiOBp7\npFB1h/LqiLi/fP4F4MCI2I6ipP0Xe5rMzMyyUjUoP0XSlIgYAmZGxKUAEXGzpOndXGBO5DUN8IKp\neU1zBnh40aapI9R21aS8tgm4mNtTR6ht2dKh1BFqW3etCb7VUuaD8lUNyteBMyQdAZwp6SjgRGBP\n4G9Wz5uZ2Tjk9/fsSqpWyn9V0jXAIcAO5fu3B04G/qP38czMLBfdbLB1PnD+qsfL/VCObT6SmdnE\nNGE22FoN74diZtakzKcNez8UMzNrhPdDMTNri0EelMf7oZiZ9U3uYyg93w9luvJqcj+16/+kjlDb\nlA3zK7n/8B+WpY5Qy9V35Lc+4vqZ+dV+vYv81s7YU/L7jjMzG1R5/f39N9ygmJm1RO5dXi5fb2Zm\njehJg9JZvv7UJbf14hJmZoNnuMFHAlXl6+dL+pWk4yVtKelsSQ9LulTSc0Y7r7N8/X6ztmk+tZnZ\nAIrh5h4pVN2hfB34PPBzinUnR0fE2sDHytfMzMyA6gZlakT8IiJOACIifkrx5Fwgv7mqZmZtlnmX\nV9Usr8clvQJYGwhJr4uIk8vdGrva6OQ5b3psvBn76mcnbZE6Qm0HfnmH1BFqW/+FS1NHqOXFl1yV\nOkJtfzht3dQRatskJvbE01RdVU2p+n/v/RRdXsMUJVgOkXQccA/w3t5GMzOznIzZ5RURV0XEKyNi\n34i4MSI+GBHrRMQzgR37lNHMbGLIvMvL5evNzFoi91leLl9vZmaNcPl6M7OWGPRBeZevNzPrk4Fu\nUJooX3/Gz/KauvhohrMWb/rwZakj1PbYE9NSR6jlsmnrpY5Q2x2TnkgdobYg7+KIE12Gvz7NzAZU\nKHWCcXGDYmbWErl3ebl8vZmZNaJq2vDawMeB1wEbAQEsAk4BjoiIh3qe0MxsgojhvLu8qu5Qfkwx\nZfhlEbFeRKwP7FEe+/FoJ3Xuh3LOkluaS2tmNsByX9hY1aDMi4gjI+IvIwci4i8RcSSw1Wgnde6H\nsves7ZrKamZmDZK0j6SbJN0i6WPj/XxVDcqdkj4i6clV8ZI2lvRR4K7xXtzMzJ4SocYeVSRNBv4v\nsC/wDOAtkp4xnvxVs7wOpNhM64KyUQngPuBU4O+7ucC+r/mf8eTruxjKb5rFtH1enDpCbcNXX5s6\nQi3Pnp7f+ohzvj41dYTatpm5OHWEpPrcVbU7cEtE3AYg6YfA/sD1a/oJqxY2/lXSscDZwEUR8ejI\na5L2Ac5c0wubmVnvSFoALOg4tDAiFnZ8vDkr9zTdDTxvPNesmuV1GPCPwA3AtyV9MCJOKV/+LG5Q\nzMwa0+Qsr7LxWFj5xgZVdXm9F9g1Ih6VNA/4qaR5EXEURYFIMzNrSPS3Z/UeYMuOj7coj62xqgZl\n0kg3V0TcIellFI3KVrhBMTPL2aXA9pK2pmhI3gx0VaNxNFWzvO6T9OyRD8rG5TXABsDO47mwmZmt\nLIbV2KPyWhFDwKHALymGNX4cEdeNJ3/VHcpBwNBqQhwk6ejxXNjMzFbW75XyEXEGcEZTn69qltfd\nY7z2224ucPGpeZWvv2hGfvUyLzv30tQRatt80qzUEWpZO8Oy6ptNza9U34PL1k4doTZ31Twlv9+e\nZmYDqs+D8o1zg2Jm1hK5F4d0g2Jm1hLdlExpszE7WSXNlfQ5Sd+X9NZVXvt6b6OZmVlOqkbtjqVY\nb/Iz4M2SfiZpevna80c7qbN8/c+X3tpQVDOzwZZ7+fqqLq9tI+KN5fOTJR0OnCdpv7FO6lzyf/bG\nB2Y+zGRm1h/DmXd5VTUo0yVNiijau4j4jKR7gAuB2T1PZ2Zm2ahqUE4D9gTOGTkQEcdJ+gvw1W4u\ncPbMvObCbz+U318I35z3WOoItX3l7jmpI9RyWyxJHaG2c1f8NXWE2rabul7qCLUd3ODnGuhB+Yj4\nCHC3pL0kze44fiZwWK/DmZlNJP0svdILVbO8PgCcAnwAuFbS/h0vf6aXwczMLC9VXV4LcPl6M7O+\nGPSV8i5fb2bWJ7mvlHf5ejMza4TL15uZtcRAr0Npony9mZl1J/dpwz0vDrnR8OReX6JRWy4fqn5T\ny/zwT5unjlDbG4byWtdx1tS1Ukeobda0vH72AG4fejh1BBsHVxs2M2uJQZ/l9TckbRQRi3oRxsxs\nIhvoMRRJq9ZBEHCJpOcAiogHe5bMzMyyUnWHcj9w5yrHNgeuAALYZnUnSVpAsSiSN667O8+fvf04\nY5qZDb7cB+Wr1qF8GLgJ2C8ito6IrYG7y+erbUygKF8fEfMjYr4bEzOz7kQ090ihqjjkF4H3AJ+Q\n9CVJcyjuTMzMzFZSOShfrkU5oNxU62xgVp0L7DPloTWMlsbcjZamjlDbvfdumjpCbWvNWJY6Qi3P\nXDIjdYTatpg8M3WE2jafOr36TQNsoAflASQ9nWLc5DyKBmXb8vg+ZRl7MzNrwECPoUg6jI7y9cAr\nIuLa8uXP9jibmZllpOoO5b24fL2ZWV8MepeXy9ebmfVJ7jOeXL7ezMwa4fL1ZmYtMdBdXi5fb2bW\nP7nP8up5teErH1+n15do1EX351emfMeqjssWunnJ3NQRarkiv2UonPr4rakj1LbP1HmpI9g4uHy9\nmVlLDKcOME5uUMzMWiIynzxbu7NE0vq9CGJmZnmrWil/hKQNyufzJd0GXCzpTkkvHeO8BZIuk3TZ\neUv+2HBkM7PBNBzNPVKoukN5dUTcXz7/AnBgRGwHvBz44mgndZav33OWy9ebmXVjGDX2SKGqQZki\naWScZWZEXAoQETcDE7ssqJmZraRqUP7rwBmSjgDOlHQUcCKwJ3Blr8OZmU0kuQ/KVy1s/Kqka4BD\ngB3K928PnAz8ZzcXeGjyeCP21/bDU1NHqG1WhgWAFk3Na/HM64YeTR2htgdmbJk6Qm03DS9OHSGp\niTBt+C/AQuDikUKRUOyHAng/FDMzA2ruhyJp/46XvR+KmVmDAjX2SMH7oZiZtcSgd3l5PxQzM+uK\n90MxM2uJ4QYfKXg/FDOzlhj0acPj3g/lRcprGuAPJs1KHaG2nVZkNjcbuGzKstQRajkr8uvdvm/5\n/dVvapnXTNksdQQbh7wWA5iZDbBhNfcYD0kHSLpO0rCk+d2e5/L1ZmYtkaoG12pcC7wBqDW04QbF\nzMxWEhE3AEj1GriqhY3zJf1K0vGStpR0tqSHJV0q6TljnPdk+fqfPXpHrUBmZhNVNPjo/D1cPhb0\nOn83xSE/CawD/A74p4h4uaS9ytdesLqTImIhRbkW/vC0/TOsNGVm1n9NTv3o/D28OpLOATZZzUuH\nR8Qpa3LNqgZlakT8orz4kRHx0zLouZL+a00uaGZm6UXE3k1/zqoG5XFJrwDWBkLS6yLi5HK3xhVN\nhzEzm8iGa45ZtE1Vg/J+4PMUd2KvBA6RdBxwD0Wdr0pPe9ZD48nXd5/cannqCLXN+NTXUkeo7cCT\n88o8dPXNqSPUdsvJc1JHqO3mZRO7h7wt/3pJrwe+CmwI/FzSlRHxyqrzqhY2XiXpQ8BmwN0R8UHg\ng+UF9xl/bDMza5uIOAk4qe553ZSvPwmXrzcz67lBr+X1XmC+y9ebmfXeeFe4p+by9WZmLdGilfJr\nxOXrzcysEVUNykEUe8o/KSKGIuIg4CU9S2VmNgE1uVI+hZ6Xr5+yTl7lwu7/dX7Thjf+6VGpI9QW\ni/Iqra4p+RXmvmQ4v2nDWwxP7OVtuY+h5PdTYmZmrZTX7YOZ2QDLbxu3lblBMTNribaslF9TYzYo\nkqYA7wZeT7FaHoqyK6cAx0REfgMOZmbWE1VjKN8Hng18CnhV+fg08Czg+NFO6qzDf9wf72koqpnZ\nYGvLFsBrqqrLa9eI2GGVY3cDF0katVpeZx3+h9++V+53cWZmfZH7GErVHcqD5Wb1T75P0iRJBwJ/\n7W00MzPLSdUdypuBI4H/K2mkDv06wK/K1yr99qwN1zxdAutPXpY6Qm0b3vqn1BFqG37osdQRahla\n9ETqCLVdP2VW6gi1rb9icuoISeV+h1K1sPEOSV8CvgjcCjydYtvf6yPi9j7kMzObMCLzhY1Vs7w+\nCexbvu9sYHfgfOBjkp4TEZ/peUIzM8tCVZfXmyhmeU2nqOm1RUQ8Uu4nfzHgBsXMrCED3eUFDEXE\nCmCJpFsj4hGAiFgqKfd/u5lZq+T+S7VqltcySSMje7uOHJS0Nvn/283MrEFVdygviYgnACKiswGZ\nCryjZ6nMzCag3BftVc3yWu1cyYi4H8ir/riZWcvlXr6+58UhX3zgo72+RKP+eFJ+8+CX3fxQ9Zta\nZsWS1AnqmbJOfj/pz1+W3/fyBsP5rQOzp7jasJlZS+Q+MO0GxcysJXJvUMac5SVpsqT3SfoPSS9a\n5bV/6200MzPLSdW04aOBlwIPAP9dlmEZ8YbRTuosX3/stfnVmTIzSyEafKRQ1aDsHhFvjYivAM8D\nZks6UdJ0YNRRyohYGBHzI2L+wTs9rcm8ZmYDK/f9UKoalGkjTyJiKCIWAFcB5wGzexnMzMzyUjUo\nf5mkfSLizJEDEfFpSfcA3+jmAveePTSefH1377I5qSPUds6V+WXe/fG8ysGvqPzbq32m5BeZO6ZM\nTx0hqYEelI+It1FssrUbgKRnSPpn4M8RMbUfAc3MJorcx1C6Ll8v6WyKcZRf4fL1Zma2CpevNzNr\nieHMq3m5fL2ZWUvk/kvV5evNzKwRLl9vZtYSeXd4uXy9mVlr5N7t0/PikLc9uE6vL9GoC2emTlDf\nWpHf3zV3T85rvcEu0x5OHaG2X5Hf+qT9l2e2r4GtxNWGzcxawhtsmZlZI3KfNly7OIOkm3sRxMzM\n8la1H8piSY+Uj8WSFgPbjhwf47wny9efsfTWxkObmQ2i3EuvVN2hHAucDGwfEXMiYg7wp/L53NFO\n6ixf/6qZ2zaZ18xsYA03+EihqjjkYcBRwAmSDpM0ifynSpuZWQ9UDspHxOWS9gYOBS4AZtS5wPP3\nXrSG0dK44debpY5Q2xbL82vjt52yOHWEgbf1UH716+8YnlX9ppbZrcHPlfugfGWDIml3ICLivyX9\nAdhD0qsi4ozexzMzmzjybk7ql6/fHTgfl683M7NVuHy9mVlLtKX0iqQvAK8FlgG3AgdHxENV51V1\nsg5FxIqIWAKsVL6e9vzbzcwGwjDR2GOczgZ2iohdgJuBj3dzksvXm5nZSiLirIgYKj+8CNiim/Oq\nGpSXlHcnLl9vZtZjTS5s7FxgXj4WrGGsdwG/6OaNLl9vZtYSTXb7RMRCYOFor0s6B9hkNS8dHhGn\nlO85HBgCftDNNXteHPLgX6/V60s06oEV96aOUNsNS+5OHaG2SeRVVnXrtVb3c9duU5VfKfhPDW+Q\nOsKEERF7j/W6pHcCrwH2iuhujwxXGzYza4loyUoUSfsAHwFeOjLs0Q03KGZmLdGimU5fo1gucrYk\ngIsi4v1VJ7lBMTNribaUXomI7dbkvKry9YdK2qB8vp2kCyU9JOliSTuPcd6TswvuePTONcllZmaZ\nqZo2fEg5owuKqsNfjoh1gI8C3xztpM7y9fNmb9VQVDOzwZb7fihVXV6dr28UEScBRMT5kub0LpaZ\n2cTTli6vNVV1h/JTScdJ2gY4SdKHJG0l6WDgT33IZ2Zmmaha2Hh4ORf5BGBbilH/BRS7OP5DNxd4\nEWuPM2J/PTA1vxuvOybntecMwP9aq6tKDq0xd9K01BFq2395ft/LwyxPHSGpFs3yWiPdzPK6Hjg0\nIi6V9ExgH+CGiHi4t9HMzCaWtqxDWVPeD8XMzBrh/VDMzFpi0Lu8hiJiBbBE0kr7oUjK/d9uZtYq\nuXd5eT8UMzNrRNUdyktGSth7PxQzs97K/a/0nu+HMv+J1X6K1lqs/MqbHf6+HVJHqG3xrx9MHaGW\n3966XuoIte04I7+JmOcP5bXMAODlDX6u4e6qxLdWVZeXmZlZV/L7c9zMbEDlfX/iBsXMrDUGupaX\npG0kfUfSf0qaLelbkq6V9BNJ8/oT0czMclA1hnIccCnwKHARcCPFyvkzge+MdlLnfiinLb2toahm\nZoMtGvxfClUNypyI+EZEHAHMjYgvRsRdEXEMsO5oJ3Xuh/Lamds0GtjMbFANN/hIoapBGZa0g6Td\ngVmS5kOxeyMwuefpzMwsG1WD8h8BTqNo8F4HfFzSLsDawHu7ucA106aPK2C/ndbd8ppWOfuE/Obu\nbxibpY5Qy++n/jV1hNrWGp6aOkJte+c9Jj1uuQ/KVy1sPFfSQcBwWb7+rxRjKNdHxBl9SWhmNkHk\nXsvL5evNzKwRLl9vZtYSA13LC5evNzPrmxjwWl4uX29mZl1x+Xozs5YY9Fle4y5f/2hm9YzfNrRB\n6gi1PZHh9+APWZQ6Qi07TcmvfP0jLE8dobbbpqxIHSGp3Lt9Mvt1b2ZmbeVqw2ZmLTHQ61DMzKx/\nBnoMRdIU4N3A64GRWhn3AKcAx0REfp20ZmbWE1V3KN8HHgI+BdxdHtuCYobX8cCBqztJ0gJgAcDr\n1tud3Wdv30RWM7OBlvs6lKoGZdeI2GGVY3cDF0m6ebSTImIhsBDgc1u9Le+vkJlZnwz6LK8HJR0g\n6cn3SZok6UAgv/KrZmbWM2YerS0AABRASURBVFV3KG8GjgS+XlYaFkXp+l+Vr1XaflleNyizh/Ob\nB//A5PzmVhw2nNd6nxhKnaC+u6fOTB2hto2H8vp90bSBnuUVEXdQjpNIWr88fFREvK3HuczMJpxB\nn+V16moO7zlyPCL260kqMzPLTlVfyRbA9cC3gaDo8toN+GKPc5mZTTi5z/KqGpSfD1wOHA48HBHn\nA0sj4oKIuKDX4czMJpJhorFHClVjKMPAlyX9pPzvfVXnmJnZxNRV4xARdwMHSHo18EhvI5mZTUwD\nPctrVRHxc+DnPcpiZjahDWc+htLz7qupmX2BFk+anDpCbS/e7N7UEWqbvWVeCzsWXbdW6gi1LXw8\ndYL6njF97dQRajsgdYAW8XiImVlL5PXn999yg2Jm1hK5L2wcc9qwpMmS3ifpPyS9aJXX/q230czM\nLCdV61COBl4KPAD8t6Qvdbz2htFOkrRA0mWSLvvlklsaiGlmNvhyX4dS1aDsHhFvjYivAM8DZks6\nUdJ0ilXzqxURCyNifkTMf+Ws7ZrMa2Y2sCKiscd4lL1SV0u6UtJZkjarPqu6QZnW8Q8diogFwFXA\necDsNY9rZmYt9oWI2CUing2cDnyim5OqBuUvk7RPRJw5ciAiPi3pHuAb3Vxgk0l5zV28dPKs1BFq\nu/GuDVNHqO2ZUxeljlDL0senpo5Q2yeG8psCv2w4r+nkTWvLoHxEdC5gX4suJ6BVlV75mzL1kr4X\nEQdRFIw0M7OGNLlSvnMr9tLCcjfdbs//DHAQ8DCwRzfn1C1fL2APSeuAy9ebmbVV51bsqyPpHGCT\n1bx0eEScEhGHA4dL+jhwKPDJqmtWdXltCVzHyuXr5+Py9WZmjetn+fqI2LvLt/4AOIMuGpSqQfld\ncfl6M7O+aMu0YUnbd3y4P3BjN+e5fL2ZWUu0aIOtIyTtCAwDdwLv7+Ykl683M7OVRMQb1+Q8l683\nM2uJtkwbXlM9777K7cvz+Kjr/9trpvKbu/+9P3e18LY19hxakjpCbSdMm1b9ppZZTH7fyy9p8HPl\nvsFW1aC8mZlZVzzAbmbWEt6x0czMGjHhurwk3dzFe54sX3/yktvXLJmZmWWlqvTKYp4aVx8Zrp41\ncjwi5q7uvM4l/xdt9oa8m1wzsz4Z9C6vY4F1gA9HxH0Akm6PiK17nszMbILJvcuraqX8YZJ2BU6Q\ndDLwNdZgJvDiFRmV/p4C73nBPalT1HLNBevzEBl9jUsbrUidoJ4blNvWBkM8Z3lew6QXTh3ieUMz\nUsfo2sVT8tqeo9cqx1Ai4nJgpIjYBUCt/7ezakzIrzEB3Jj0QX6NCdk1JkBWjQk0n3c4orFHCl0N\nykfEcET8N/D3wPTeRjIzm5iiwf+lUHc/FIDpI8e9H4qZmY2ouifeArielfdD2Q3vh2Jm1rjcZ3lV\ndXnNx/uhmJn1xUB3eXk/FDMz65b3QzEza4nib/h89Xw/lMeVV0Hjvzt/aeoItR04I6+vMcDMzLqK\nfzG8KHWE2iZn9rMHsNPkdVJHSCr3/VDy+44zM7NW8niImVlLtGhP+TXiBsXMrCUGustL0qGSNiif\nbyfpQkkPSbpY0s5jnPdk+fozl9zSdGYzM2uhqjGUQyLi/vL5UcCXI2Id4KPAN0c7KSIWRsT8iJi/\nz6ztGopqZjbYIqKxRwpVXV6dr28UEScBRMT5kub0LpaZ2cQz6CvlfyrpOEnbACdJ+pCkrSQdDPyp\nD/nMzCwTVSvlDy8bjxOAbSkqDS8ATgb+oZsLLJoyebwZ++qD5NdFt/3jT6SOUNvGcx9LHaGWbRZv\nkDpCbVtPy+trDPCDJzeGnZgGeoMtgIg4lmLnRgAkfT8i/rWnqczMJqCBnjY8Svn6PV2+3szMVuXy\n9WZmLTHQ61Bw+Xozs74Z6GnDLl9vZmbdcvl6M7OWyH0dSs/L1587Oa+pi1977oOpI9S2+M6pqSPU\ndu9f5qaOUMtfpuZXmPucSdNTR6htGctTR0gq91le+f2UmJlZK3k8xMysJXKf5eUGxcysJQa6y0vS\nNpK+I+k/Jc2W9C1J10r6iaR5/YloZmY5qBpDOQ64FHgUuAi4EdgXOBP4zmgnde6HcsujdzST1Mxs\nwA1HNPZIoapBmRMR34iII4C5EfHFiLgrIo4B1h3tpM79ULabPa/JvGZmAysa/F8KVQ3KsKQdJO0G\nzJI0H4rdG4G8ygibmVlPVQ3KfwQ4DRgGXgd8XNIuwNoUZewrfXbdR8cVsN9+cfGWqSPUtoPyWusD\n8LR5f00doZZ5U/Jbn3THnzZNHaG2TYenpY6Q1EAvbIyIc4EdOw79RtLpwH5lWRYzM2tI7rO81qR8\n/cuAkyW5fL2ZmT2pqstrS+A6XL7ezKznct+xsWpQfldcvt7MrC9cvt7MzAyXrzcza42BHpRf1ZqU\nrzczs+7k3ZyAcm4RJS2IiIWpc3Qrt7yQX+bc8oIz90NueXOV+34oXS2ubJHc8kJ+mXPLC87cD7nl\nzVLuDYqZmbWEGxQzM2tE7g1Kbn2iueWF/DLnlhecuR9yy5ulrAflzcysPXK/QzEzs5Zwg2JmZo1w\ng2JmZo1wg2JmZo3IqkGRtLGkYyT9ovz4GZLenTrXIJG0lqRJ5fMdJO0naWrqXGOR9MFujrWJpM9L\nmitpqqRzJf2PpLelzjUWSed2c6wtJP37Kh9PlvSDVHkmgqwaFOA44JfAZuXHNwMfSpamC+Uv5W9J\nOkvSeSOP1LnGcCEwQ9LmwFnA2ym+7m32jtUce2e/Q9T0ioh4BHgNcAewHfDhpIlGIWmGpPWADSSt\nK2m98jEP2DxtujFtKenjAJKmAycCf0wbabDlVop+g4j48cg3SUQMSVqROlSFnwDfBL4FtD0rFFPJ\nl5R3fl+PiM9LujJ1qNWR9BbgrcDWq+wuOgdo+ybwI3d9rwZ+EhEPS0qZZyzvo/jDbTOK/ZFGgj4C\nfC1VqC68C/hB+ftiD+CMiPhK4kwDLbcG5TFJ61MW5ZT0fODhtJEqDUXEN1KHqEGSXgD8AzDSnTg5\nYZ6x/A64F9iAlXcRXQxcnSRR906TdCOwFDhE0obA44kzrVZEHAUcJekDEfHV1HmqSHpux4dHAUcD\nvwUulPTciLgiTbLBl9XCxvIb5avATsC1wIbAmyKitb88JH0KWAScBDwxcjwiWvkXtKSXAv8C/DYi\njpS0DfChiDgscbSBUnbBrEWxE+oKSWsBsyPivsTRRiXpP4BPRcSK8uO5wFERcXDaZCuT9KsxXo6I\n2LNvYSaYrBoUAElTgB0pbrtviojliSONSdLtqzkcEbFN38MMKElvAI4ENqL4vhDF13hu0mBjkHRF\nRDy36libSPos8ErgYGBjiu6ur0ZEm7u9rI9ybFBeCMyjo7suIr6XLNCAkPSViPiQpNNYzT4/EbFf\nglhdkXQL8NqIuCF1liqSNqEYyD6eYvxnZDxiLvDNiHh6qmzdkLQXcDrwV+AlEXFL4kijkrQx8Flg\ns4jYV9IzgBdExDGJow2srBoUSd8HtgWu5KkB7mhzd0w55fYQ4CXlofOBo9t2ZyVp14i4vOzy+hsR\ncUG/M3VL0m8j4kWpc3RD0jsoZqDNBy5l5QHu70bEiYmiVZL0EuAbFI3hzsC6wLsj4s9Jg42iXF5w\nLHB4RDyr7N34Q0TsnDjawMqtQbkBeEZkFFrStylm9Hy3PPR2YEVEvCddqsEi6ShgE+BkVh6nauUv\n53Kdz1siIqs1EZIuAd4ZEdeXH78B+Gxb76okXRoRu0n6Q0Q8pzx2ZUQ8O3W2QZXbLK9rKX5x3Js6\nSA27RcSzOj4+T9JVydJUkPQi4FPAVhTfHyPjEW0e85kLLAFe0XEsKNYdtE5EDEv6JyCrBoWiu+jJ\nqe8RcaKk1t65kues0Kxl0aB09OvPAa4v/1Lq/Eu0tf37wApJ20bErQDlrKk2r0c5BvgnivUGbc75\npLbNMurSOZL+D/Aj4LGRg22d/VfaVtI3gI0jYidJuwD7Af+ZONdo/hk4lSL3bylnhaaNNNiy6PIq\n+/VFMZPnI50vAUdGxPOSBOtCOYh5LHAbRd6tgIMjYqypjclIurjNX8/VkbQDRd/+Sr/oIqKtv+iy\nnP1X3o18mGIMcKQL6dqI2CltstHlNis0d1k0KCNGmWp5dUTskipTN8o1BzuWH94UEU+M9f6UJB1B\nsZDxRFa+C2ztYrAcf9HlKLcxCUmzKO5StoqI90raHtgxIk5PHG1g5dLldQjwv4FtJHUuYpxDsQK2\ndcoBy9XZTlJrB4yBkbuT+R3HAmjzYrBZEXHJKqVLhlKF6ZaknYBnADNGjrV8Cvz9krblqTGJN9Hu\n8cxjKbpuX1B+fA9FKSQ3KD2SRYMC/D/gF8DngI91HF/c4j7n15b/3Qh4IXAuxW33HhQlQ1rZoETE\nHqkzrIHcftEh6ZPAyygalDOAfYHfAG1uUP6RYm/2p0u6B7idokRPW20bEQeWNd8oa9S1tmDaIMii\nQYmIhylmZ7wldZZujQwUSzqLYqrzveXHm9Li6r1l99wb+dvFo/8+2jktsLpfdK0uBU8xOPwsinUR\nB5eL8I5PnKnKPRR/9f8KWI9i7cw7gLZ+byyTNJOn/tDYlo5uXGteFg1K5rYcaUxK9wFPSxWmC6dQ\nNN6Xk8kPX0TcBuxd1sOaFBGLU2fqwtJy+vBQWRNrEbBl6lAVTgEeAq4AWrmYcRWfBM6kKGP/A+BF\ntH9bg6y5Qem9cyX9Ejih/PhA4JyEeapsERH7pA5Rh6R1gIMo76pGejXaXEEBuKzM/S2KxvtR4Pdp\nI1XK7XvjHcDPgZ9SzLL8YETcnzbSYMtqlleuJL2ep0qvXBgRJ6XMMxZJCykK/l2TOku3JP0OuAi4\nBhgeOR4R3x31pBZRsVHV3DZXzYb8vjck7QG8uHxsC/yB4ufvqKTBBpgblD6QtBWwfUScU05lnNzW\nbhlJ11PsHng7RZfXyEr51k7NbnuV3tWRdG5E7FV1rE0y/d6YDOxGMRnm/RRdja0sFTMI3OXVY5Le\nCyygGMTclqLS7DeBtv7i2Dd1gDXw/fLrfDot33NG0gxgFuV2uqxcbbjN2+lCZt8bKva7X4uiK/HX\nFGWQFqVNNdjcoPTePwK7AxcDRMQfJW2UNtLoIuJOSX9HcUd1rIqdBGenzlVhGfAF4HCeKr0fQBtX\nna9uO92g2GWy1bshRsSdqTPUdDWwK8WGfA8DD0n6fUQsTRtrcE1KHWACeCIilo18UJaCaG0/Y7k+\n4qPAx8tDU2n/dNZ/AbaLiHkRsXX5aGNjQkQcFRFbA58Bnl0+HynN0/ZB+axExD9FxEuANwAPUHyd\nH0qbarC5Qem9CyT9KzBT0sspVuqeljjTWF5PUfDvMYByr4s5SRNVu4Wi2nBO3hQRj5R3g3sC36ao\nR2YNkXSopB9RDMbvD3yHzLrtcuMur977GPBuihlI76NYFf3tpInGtiwiQtLIYrC1UgfqwmPAlSr2\nEu8cQ2nztOGRSs6vBr4VET+X1NpilpmaAXwJuDwiWl+KZxB4lpetpCypvj3wcopSN+8C/l9EtLZ/\nv9wF8W+0edqwpNMpVp6/HHgusBS4ZJW9c8yy4galRyRdwxhjJW2dainpSIqFl6+gGDD+JbB3RHw0\nabABU04f3we4ppyosSmwc0SclTia2Rpzg9Ij5dqTUbV1xkyOWwSUZck/x99W7m3lwLzZoPIYSo90\nNhiSNqGYOhzApRHxl2TBRpHjFgEdjqWo2/RligVsB+MJJ2Z95zuUHpP0HuATwHkUXUgvBf49Ir6T\nNNgqJK0NrEteWwQAIOnyiNhV0jURsXPnsdTZzCYS36H03oeB50TEAwCS1qfYD6VVDUqOWwR0eELS\nJOCPkg6lGOxu+2JMs4HjboHee4BiFfSIxeUxa84HKcqZHEaxMvrtFJVmzayP3OXVI5L+uXz6bGBn\nir0kgmKB1dUR8c5E0czMesJdXr0zsrr81vIx4pQEWQaapB0ouha3YuVdJvdMFspsAvIdimVP0lUU\nFZwv56kV6ETE5clCmU1AblB6TNJ8iiq4q/713Np1HbnxjC6zdnCD0mOSbqLojll1N8FWLmzMkaRP\nUezJfhIt3w/FbJC5QekxSb+JiL9LnWOQSbp9NYfDK+XN+ssNSo9J2otibce5rPzX84nJQg2Qcv3J\nARHxo9RZzCY6Nyg9Jul44OnAdTzV5RUR8a50qQaLpMsiYn7qHGYTnRuUHpN0U0TsmDrHIJN0BHA/\n8CPKjcHAYyhm/eYGpcckHQt8ISKuT51lUHkMxawd3KD0mKQbgG2B2ynGUETxy87Ths1soLhB6bHR\n9kXxtOHmSDpodccj4nv9zmI2kbn0So9ImhsRj7ByYUjrjd06ns8A9gKuANygmPWR71B6RNLpEfGa\nsn8/KLq6Rrh/v4ckrQP8MCL2SZ3FbCLxHUqPRMRryqe/BS4Afh0RNyaMNJE8BmydOoTZROMGpfeO\nAV4MfFXSthRdMb+OiKPSxhockk6juAsEmAz8L+DH6RKZTUzu8uoDSZMp+vn3AN4PLI2Ip6dNNTgk\nvbTjwyHgzoi4O1Ues4nKDUqPSToXWAv4PfBr4DcRsShtqsEjaWOeGpy/xF9js/7zFsC9dzWwDNgJ\n2AXYSdLMtJEGi6S/By4BDgD+HrhY0pvSpjKbeHyH0ieS5gDvBP4PsElETE+baHCUG2y9fOSuRNKG\nwDkR8ay0ycwmFg/K95ikQykG5XcF7gC+Q9H1Zc2ZtEoX1wP47tus79yg9N4M4EvA5RExlDrMgPqF\npF8CJ5QfHwickTCP2YTkBqXHIuK/UmeYAAI4GhjZyGwh8Px0ccwmJo+hWPYkXRERz13l2NUuwGnW\nX75DsWxJOgT438A2kq7ueGkORYUCM+sj36FYtiStDawLfA74WMdLi725lln/uUExM7NGeGqlmZk1\nwg2KmZk1wg2KmZk1wg2KmZk14v8Dubmt4eP0a/YAAAAASUVORK5CYII=\n","text/plain":["<Figure size 504x504 with 2 Axes>"]},"metadata":{"tags":[]}}]},{"cell_type":"markdown","metadata":{"id":"cTHPBDzWPI8s","colab_type":"text"},"source":["1D global max-pooling would extract the highest value from each of our num_filters for each filter size. We could also follow this same approach to figure out which n-gram is most relevant but notice in the heatmap above that many filters don't have much variance. To mitigate this, this [paper](https://www.aclweb.org/anthology/W18-5408/) uses threshold values to determine which filters to use for interpretability. \n","\n","But to keep things simple, let's extract which tokens' filter outputs were extracted via max-pooling the most frequenctly. "]},{"cell_type":"code","metadata":{"id":"P72CZhU0CtGa","colab_type":"code","outputId":"d6d177de-6aea-4ec3-cc0b-1a7fb8a806df","executionInfo":{"status":"ok","timestamp":1584551640663,"user_tz":420,"elapsed":978,"user":{"displayName":"Goku Mohandas","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjMIOf3R_zwS_zZx4ZyPMtQe0lOkGpPOEUEKWpM7g=s64","userId":"00378334517810298963"}},"colab":{"base_uri":"https://localhost:8080/","height":136}},"source":["sample_index = 0\n","print (f\"Preprocessed text:\\n{decode(indices=X_infer[sample_index], tokenizer=X_tokenizer)}\")\n","print (\"\\nMost important n-grams:\")\n","# Process conv outputs for each unique filter size\n","for i, filter_size in enumerate(FILTER_SIZES):\n","\n","    # Identify most important n-gram (excluding last token)\n","    popular_indices = collections.Counter([np.argmax(conv_output) \\\n","            for conv_output in conv_outputs[i][0]])\n","    \n","    # Get corresponding text\n","    start = popular_indices.most_common(1)[-1][0]\n","    n_gram = \" \".join([token for token in tokens[start:start+filter_size]])\n","    print (f\"[{filter_size}-gram]: {n_gram}\")"],"execution_count":137,"outputs":[{"output_type":"stream","text":["Preprocessed text:\n","the wimbledon tennis tournament starts next week\n","\n","Most important n-grams:\n","[2-gram]: week\n","[3-gram]: tennis tournament starts\n","[4-gram]: the wimbledon tennis tournament\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"kbZPYQ2TH1Jt","colab_type":"text"},"source":["---\n","Share and discover ML projects at <a href=\"https://madewithml.com/\">Made With ML</a>.\n","\n","<div align=\"left\">\n","<a class=\"ai-header-badge\" target=\"_blank\" href=\"https://github.com/madewithml/lessons\"><img src=\"https://img.shields.io/github/stars/madewithml/lessons.svg?style=social&label=Star\"></a>&nbsp;\n","<a class=\"ai-header-badge\" target=\"_blank\" href=\"https://www.linkedin.com/company/madewithml\"><img src=\"https://img.shields.io/badge/style--5eba00.svg?label=LinkedIn&logo=linkedin&style=social\"></a>&nbsp;\n","<a class=\"ai-header-badge\" target=\"_blank\" href=\"https://twitter.com/madewithml\"><img src=\"https://img.shields.io/twitter/follow/madewithml.svg?label=Follow&style=social\"></a>\n","</div>\n","             "]}]}